APPENDIX
A
District Interview Protocol
TSL DISTRICT INTERVIEW PROTOCOL
Program name:
District Name:
Date:
Person interviewed:
Name(s):
Title(s):
Phone number:
Email:
Interviewer name:
Before the interview (email)
Prior to the interview, e-mail the respondent with the following text:
We are contacting you because the U.S. Department of Education wants to learn more about how TSL grantees are using their grants to support educators. You were selected to participate in the interview because your district has received funds from the TSL grant program. ED regulations require recipients of federal funds to cooperate in evaluations of grant programs under which they receive funding (Education Department General Administrative Regulations Section, 34 C.F.R. § 76.591).
Grantees use TSL funds to support and improve their human capital management and performance-based compensation systems in a variety of ways. All aspects of this data collection will help us describe TSL grantees’ goals in using their grants to improve educator effectiveness, as well as the strategies they are using to accomplish these objectives. The total time our interview will take is an hour. We’d like to divide that time into two calls:
During the first call, we’d like to learn about the activities your grant funds, how much of your total grant budget has been allocated to each actvitiy, and the strategic priority of the activity for accomplishing the goals of the grant. Our goal will be to identify the three highest priority activities funded by your TSL grant. We expect the first call to take about 20 minutes of your time. Below we’ve provided a list of activities that might be funded by your TSL grant. We’ve grouped the activities into six somewhat subjective categories, so please pay more attention to the activities than the categories. Prior to our first call, please review the list of activities. When we talk, we will ask about which activities your district uses TSL funds to undertake, the amount of allocated funds for each activity, and each activity’s strategic priority. Reviewing the list prior to the call and perhaps making any notes for yourself about funding and strategic priority will help ensure that the first call will accomplish our goal in the expected time.
During the second call, we will focus on learning more about the three highest priority activities identified in the first call. We expect the second call to take about 40 minutes of your time.
Please suggest several dates and times that might work for the first and second calls. We look forward to hearing about the important work your district is doing as part of the TSL grant.
Sincerely,
XXXXX
Please review this list of activities that might be funded by your TSL grant. For any activities you do as part of the TSL grant, you may want to make notes for yourself about the funding amount and strategic priority for your grant. When we talk, we will ask about which activities your district uses TSL funds to undertake, the amount of allocated funds for each activity, and each activity’s strategic priority.
Recruiting a high-quality and diverse workforce
Developing partnerships with teacher prep programs (e.g. a teacher residency program)
Providing incentives to attract minority of bilingual teachers to work in the district or at particular schools
Revise teacher or principal recruitment practices
Supporting improvement of teachers or school leaders
Improving or developing new teacher induction activities
Providing general professional development for teachers or school leaders
Providing additional tools or resources focused on instructional improvement to teachers or school leaders (e.g., curriculum kits or subscriptions)
Providing targeted professional development based on teacher or school needs
Providing additional support for teachers (e.g., support from a colleague or hiring new mentors
Retaining and rewarding high-performing educators
Providing additional pay to teachers or school leaders based on their performance, without requiring extra work (e.g., performance bonuses)
Providing additional pay to educators for taking on extra duties (e.g., teacher leaders)
Providing additional pay for high-performing educators who transfer to high-need schools
Providing additional pay for high-performing teachers to teach hard-to-staff subjects
Providing additional opportunities or career pathways for high-performing educators without additional pay
Improving or creating measures to assess educator effectiveness used to inform performance-based compensation decisions, such as educator incentives or stipends
Revising how to measure or calculate overall educator effectiveness
Improving or revising teacher or principal observation measures
Improving or revising student achievement growth measures
Generating a new measure of educator effectiveness
Improving or creating other meaures aside from educator effectiveness measures used to inform human capital management decisions, such as teacher recruitment strategy
Revising existing measures (e.g., student or teacher attendance, student behavior, student characteristics, teacher retention)
Creating new measures
Improving data systems to make more efficient and user-friendly or incorporate new data
Converting paper records to electronic records stored in system
Incorporating new or additional data on students, teachers, or leaders into the district’s human capital management system
Revising system so it can link together different types of data
Creating data management strategies or tools (e.g., data manual, data dictionary, data use protocols)
Improving data accessibility
Creating or updating software
Any other activities funded by TSL grant
Interview 1
A. Introduction
This is _________ from Mathematica. The U.S. Department of Education has contracted with Mathematica to learn more about how TSL grantees use grant funds to develop, implement, improve or expand their human capital management systems or performance-based compensation systems for educators. During this call, we’ll be asking both about the amount of grant funds used for each activity and the activity’s priority level.
Before we get started, did you have any questions about the information we included in the e-mail about how we will use the information you provide to us? If “no”, skip this bulleted list. If “yes”, respond using this information as appropriate.
Your responses will not affect your grant—now or in the future.
We will keep your responses confidential. We will not link the names of any particular individual, school or district with any of the findings reported from this study.
An IRB has reviewed and approved this study. If you have any questions about your rights as a research participant, you can contact the survey director, Eric Zeidman, at 609-936-2784 or [email protected]).
To ensure that we are making the most of the time you have available, we are breaking this interview into two separate phone calls. This first call should take about 20 minutes and is designed to help us learn more about your district’s primary activities under your TSL grant. A 40 minute follow-up interview will be a more targeted discussion based on what I learn today.
If it is OK, I would like to record the interview. I may refer to it if I need to check a response and write up my notes. It will not be shared with anyone beyond the Mathematica study team. Would it be OK if I record this interview?
Do you have any questions? Let’s get started.
B. General information on key activities funded by TSL grant
I’m going to list some of the potential activities that districts might be doing using TSL funds. These are the same activities listed in the email we previously sent you. We’ve grouped these activities under this category somewhat arbitrarily, so please pay attention to the activity and whether you do it using TSL funds. For each activity your district does, please let me know the amount (or percentage) of funding and whether the activity is a high, moderate, or low priority in terms of the goals of your TSL grant overall.
B1. Recruit high-quality and diverse workforce
First I’m going to list some of the potential activities associated with what we’re thinking of as recruiting a high-quality and diverse workforce that you might be doing using TSL funds. We can discuss the purpose of the activity later (especially if it is not to recruit high quality and diverse workforce).
Table B1 – Activities under “recruit high-quality and diverse workforce”
Are you using the TSL grant to: |
Check box if yes |
Amount of funding for the activity or percentage (**if they provide a percentage, be sure to get the total amount of grant funds to the district) |
Indicate priority level |
1. develop partnerships with teacher prep programs (e.g. a teacher residency program) |
|
$ |
High priority Moderate priority Low priority |
2. provide incentives to attract minority or bilingual teachers either to the district or to particular schools |
|
$ |
High priority Moderate priority Low priority |
3. revise teacher or principal recruitment practices |
|
$ |
High priority Moderate priority Low priority |
Interviewer: use this space to write notes about any activities described.
B2. Support improvement of teachers or school leaders
Now I’m going to list some of the potential activities associated with what we’re thinking of as supporting improvement of teachers or school leaders that you might be doing using TSL funds. We can discuss the purpose of the activity later (especially if it is not to support improvement of teachers or school leaders).
Table B2 – Activities under “support improvement of teachers or school leaders”
Are you using the TSL grant to: |
Check box if yes |
Amount of funding for the activity or percentage (**if they provide a percentage, be sure to get the total amount of grant funds to the district) |
Indicate priority level |
1. improve or develop new teacher induction activities |
|
$ |
High priority Moderate priority Low priority |
2. provide general professional development for teachers or school leaders |
|
$ |
High priority Moderate priority Low priority |
3. provide additional tools or resources focused on instructional improvement to teachers or school leaders (e.g. curriculum kits or subscriptions) |
|
$ |
High priority Moderate priority Low priority |
4. provide targeted professional development based on teacher or school needs |
|
$ |
High priority Moderate priority Low priority |
5. provide additional support for teachers (e.g. providing additional support from a colleague or creating or hiring new mentor positions) |
|
$ |
High priority Moderate priority Low priority |
Interviewer: use this space to write notes about any activities described.
B3. Retain and reward high-performing educators
Now I’m going to list some of the potential activities associated with what we’re thinking of as retaining and rewarding high-performing educators that you might be doing using TSL funds. We can discuss the purpose of the activity later (especially if it is not to retain and reward high-performing educators).
Table B3 – Activities under “Retain and reward high-performing educators”
Are you using the TSL grant to : |
Check box if yes |
Amount of funding for the activity or percentage (**if they provide a percentage, be sure to get the total amount of grant funds to the district) |
Indicate priority level |
1. provide additional pay to teachers or school leaders based on their performance, and without requiring extra work (e.g. performance bonuses, for earning National Board Certification, salary increases based on performance) |
|
$ |
High priority Moderate priority Low priority |
2. provide additional pay to teachers or school leaders for taking on extra duties (e.g. stipend for being a teacher-leader) |
|
$ |
High priority Moderate priority Low priority |
3. provide additional pay for high-performing teachers or school leaders to transfer to high-need schools |
|
$ |
High priority Moderate priority Low priority |
4. provide additional pay for high-performing teachers to teach hard-to-staff subjects |
|
$ |
High priority Moderate priority Low priority |
5. provide additional opportunities or career pathways for highly effective teachers or school leaders, without additional pay |
|
$ |
High priority Moderate priority Low priority |
Interviewer: use this space to write notes about any activities described.
B4. Improve or create measures to assess educator effectiveness used to inform performance-based compensation decisions
Now I’m going to list some of the potential activities associated with what we’re thinking of as improving or creating measures to assess educator effectiveness used to inform performance-based compensation decisions that you might be doing using TSL funds. We can discuss the purpose of the activity later (especially if it is not to improve or create measures to assess educator effectiveness used to inform performance-based compensation decisions).
Table B4 – Activities under “Improve or create measures to assess educator effectiveness used to inform performance-based compensation decisions”
Are you using the TSL grant to: |
Check box if yes |
Amount of funding for the activity or percentage (**if they provide a percentage, be sure to get the total amount of grant funds to the district) |
Indicate priority level |
1. revise how to measure or calculate overall educator effectiveness |
|
$ |
High priority Moderate priority Low priority |
2. improve/revise teacher or principal observation measure |
|
$ |
High priority Moderate priority Low priority |
3. improve/revise student achievement growth measure |
|
$ |
High priority Moderate priority Low priority |
4. generate a new or additional measure of educator effectiveness |
|
$ |
High priority Moderate priority Low priority |
Interviewer: use this space to write notes about any activities described.
B5. Improve or create other measures aside from educator effectiveness measures used to inform other human capital management decisions
Now I’m going to list some of the potential activities associated with what we’re thinking of as improving or creating other measures aside from educator effectiveness measures used to inform other human capital management decisions that you might be doing using TSL funds. We can discuss the purpose of the activity later (especially if it is not to improve or create other measures aside from educator effectiveness measures used to inform other human capital management decisions).
Table B5 – Activities under “Improve or create other measures aside from educator effectiveness measures used to inform other human capital management decisions”
Are you using the TSL grant to: |
Check box if yes |
Amount of funding for the activity or percentage (**if they provide a percentage, be sure to get the total amount of grant funds to the district) |
Indicate priority level |
1. revise an existing measure (Measures might include student or teacher attendance, student behavior, student characteristics, teacher retention) |
|
$ |
High priority Moderate priority Low priority |
2. create a new measure (Measures might include student or teacher attendance, student behavior, student characteristics, teacher retention) |
|
$ |
High priority Moderate priority Low priority |
Interviewer: use this space to write notes about any activities described.
B6. Improve data systems to make more efficient and user-friendly or incorporate new data
Now I’m going to list some of the potential activities associated with what we’re thinking of as improving data systems to make more efficient and user-friendly or incorporate new data that you might be doing using TSL funds. We can discuss the purpose of the activity later (especially if it is not to improve data systems to make more efficient and user-friendly or incorporate new data).
Table B6 – Activities under “Improve data systems to make more efficient and user-friendly or incorporate new data”
Are you using the TSL grant to: |
Check box if yes |
Amount of funding for the activity or percentage (**if they provide a percentage, be sure to get the total amount of grant funds to the district) |
Indicate priority level |
1. convert paper records to electronic records stored in system |
|
$ |
High priority Moderate priority Low priority |
2. incorporate new or additional data on students, teachers, or principals in the district’s human capital management system |
|
$ |
High priority Moderate priority Low priority |
3. revise system so it can link together different types of data |
|
$ |
High priority Moderate priority Low priority |
4. create data management strategies or tools (e.g. data dictionary, data manual, data use protocols) |
|
$ |
High priority Moderate priority Low priority |
5. improve data accessibility |
|
$ |
High priority Moderate priority Low priority |
6. create or update software |
|
$ |
High priority Moderate priority Low priority |
Interviewer: use this space to write notes about any activities described.
Are there any other activities that you use TSL grant funds for that we did not cover? If so, please describe the activity, the amount or percentage of funds used for it, and the priority level of the activity.
TSL grant funds used to: |
Check box if yes |
Amount of funding for the activity or percentage (**if they provide a percentage, be sure to get the total amount of grant funds to the district) |
Indicate priority level |
[describe activity] |
|
$ |
High priority Moderate priority Low priority |
Wrap-up
We want to be sure that in our 40-minute follow up interview we learn more about the 3 most-important activities that your TSL grant is supporting. We want to prioritize activities that you indicated had a high amount of TSL funds allocated towards them, as well being a high priority in terms of the goals of your grant. (Review the list of activities that the respondent noted had high funding amounts AND were “high-priority”. If fewer than three are both high funding and high priority, go to activities that the respondent reported high on funds and moderate on strategic importance.)
Which would you say are the top three?
If a date/time for the follow-up interview hasn’t been scheduled already, try to do that now.
Thank you for your time. I will use the information you provided today to create a list of questions for these activities that we will use for the follow-up interview. Before that call, I will email you the list of the three activities we’ll discuss.
Interview 2
Introduction
This is _________ from Mathematica.
Thank you again for participating in our earlier call to help identify your district’s highest priority activities for your TSL grant based on the amount of funds going towards the activities and their importance in terms of your grant goals. Today’s call will take around 40 minutes and is focused on those three actvities we identified during our last call. The purpose of our conversation today is to get a deeper/richer picture of how you use the TSL grant funds to support these three activities.
TSL grants are intended to support the district’s human capital management system, but in particular, to support high-need schools. Throughout this interview we will be interested in knowing how funds are being used generally, and when appropriate, when funds are targeted towards high-need schools. We will also want to be clear which activities funded by TSL are targeted towards teachers or school leaders.
Before we get started, did you have any questions about the information we included in our initial e-mail about how we will use the information you provide to us? If “no”, skip this bulleted list. If “yes”, respond using this information as appropriate.
Your responses will not affect your grant—now or in the future.
We will keep your responses confidential. We will not link the names of any particular individual, school or district with any of the findings reported from this study.
An IRB has reviewed and approved this study. If you have any questions about your rights as a research participant, you can contact the survey director, Eric Zeidman, at 609-936-2784 or [email protected]).
If it is OK, I would like to record the interview. I may refer to it if I need to check a response and write up my notes. It will not be shared with anyone beyond the Mathematica study team. Would it be OK if I record this interview?
Let’s get started.
C. Detailed information on up to the top 3 high-priority activities
This section will probe on the three activities identified in Call 1. The goal is to get more detailed information on these activities. This protocol should be revised as necessary to focus questions and probes on the three activities identified prior to Call 2.
We want to be sure that we learn more about the three activities that your TSL grant is supporting that we identified on our earlier call. As I noted in the e-mail, today we plan to focus on (insert three activities here). Is there another activity that you think we should be discussing instead (given the funds going towards it and its importance for achieving your grant goals)? If so, please explain.
For respondents who indicate that at least 1 of their 3 activities for their TSL grant is: Recruit high-quality and diverse workforce, Support improvement of teachers or school leaders, and/or Retain and reward high-performance educators, you should cover five sections of follow-up questions:
The first section (e.g. C1.1) asks whether it is a new activity or an extension/revision of an existing activity.
The second section, (e.g. C1.2) asks the grantee to describe the activity and why they chose to make this a high-priority activity for their TSL funds. It includes specific probes for different activities.
The third section (e.g. C1.3) asks about their use of educator effectiveness data, if they used it for the activity, and if they revised how they assessed educator effectiveness to accomplish the task.
The fourth section (e.g. C1.4) focuses on their use of data, aside from educator effectiveness data, and if they improved the measure or created a new measure.
The fifth section (e.g. C1.5) asks if they revised or improved their data systems to accomplish the activity.
The final section (e.g. C1.6) asks about their successes and challenges implementing this activity.
For respondents who indicate that at least 1 of their 3 activities for their TSL grant is Improve or create measures to assess educator effectiveness used to inform performance-based compensation decisions, Improve or create other measures aside from educator effectiveness measures used to inform other human capital management decisions, and/or Improve data systems to make more efficient and user-friendly or incorporate new data, you should cover three sections of follow up questions:
The first section focuses on who the measure assesses (if appropriate – C4 and C5) (Note that we are not asking whether the activity is new or revising/expanding because that is part of the description of the activity itself (i.e. generate new measure of effectiveness).
The second section asks the grantee to describe the activity and why they chose to make it a high-priority activity for their TSL funds
The third section asks about their successes and challenges implementing the activity.
Let’s start with (Activity 1 – fill in activity), then we’ll ask about (Activity 2 - fill in activity), and end with (Activity 3 – fill in activity).
C1. Recruit high-quality and diverse workforce
One of the activities you mentioned on our last call was [insert activity] [Interviewer briefly summarize any explanation about the activity that the respondent provided on the last call.] Is my summary accurate? If yes: Great, I’d like to ask you a few questions about the activity. If no: Thanks for noting that. Can you please describe the activity?
C1.1. New/Revised activity.
Ask questions in Table C1 for the relevant activity (e.g. C1b). Note that the respondent should answer whether an activity is new under the TSL grant. If the activity was new under another grant (i.e. TIF), then we consider that activity to be an extension or revision. If you are not sure if it is new vs extended/revised, make notes about what the district was doing previously as well as the activity funded by the TSL grant and ask project leadership.
Table C1. Whether activities under “recruit high-quality and diverse workforce” are new or revised/extended
Activity |
Is this a new activity for the district, or an extension or revision of an existing activity? |
a. Develop partnerships with teacher prep programs (e.g. a teacher residency program) |
New activity Extension/ revision |
b. Provide incentives to attract minority or bilingual teachers to work in the district or at particular schools |
New activity Extension/ revision |
c. Revise teacher or principal recruitment practices |
New activity Extension/ revision |
d. If “Other” fits here, (describe): |
New activity Extension/ revision |
C1.2. Specific questions about the activity.
For the high priority activity, first ask generally about the program. Probe on the specific follow-up questions under the relevant activity (e.g. C1b), as appropriate.
Please tell me generally about the activity. (Interviewer can use information collected from first call to prompt the respondent.)
Can you tell me why this activity is a high-priority activity for your TSL grant? (General probes for any activity: Why did your district choose this particular activity to focus on? What is the goal? How do you expect this program will help you achieve this goal?)
Follow-up questions specific to each activity (if not mentioned when answering the general questions above):
Partnerships with teacher prep programs (C1a)
What teacher prep program do you partner with?
Are there other partners?
What is the district’s role in the partnership?
What kind of teachers is the partnership intended to provide? (e.g. Are you looking to improve the quality of newly hired teachers in general? Or looking to fill hard-to-staff subjects/grades? Or looking to hire effective teachers in particular high-needs schools?)
Provide incentives to attract minority or bilingual teachers to work in the district or at particular schools (C1b)
What are the incentives? How much are the incentives and do they vary? If they vary, how do they vary and based on what?
Who is eligible to receive the incentives?
How are you targeting/recruiting these teachers?
Are there requirements for those who receive incentives, such as teaching in a high-needs schools for a certain length of time or maintaining a certain level of effectiveness?
What led your district to decide to focus on providing incentives to this group of teachers?
Revise teacher or principal recruitment practices (C1c)
What has the main change to recruitment practices been? How did the district go about executing the revised approach?
Have you revised how you identify and reach out to potential teachers? If so, how?
How you revised whether and how you use incentives to attract new teachers? If so, how?
How did the district determine what to revise?
C1.3. Use of educator effectiveness data to accomplish the activity.
This next set of questions focuses on learning more about whether and how you may have used educators’ effectiveness measures to either choose to focus on this activity or to support the activity in some way. We will also ask whether you have improved or revised the effectiveness measures for this activity. A separate set of questions we ask later will cover other types of data (like data on student attendance or behavior).
First ask generally about their use of educator effectiveness data:
Did you analyze, review, or use educator effectiveness data to help you choose to focus on this activity or to support the activity in any way? If “No”, go to Question C1.4.
Please explain (1) what educator effectiveness data you are using (or used), (2) how you are using them, (3) if improving the measures was part of this effort (and if so, how they were revised), and (4) if the TSL grant funds were used at all to help revise or improve the measures.
Follow-up probes if not mentioned when answering earlier questions:
Ask if used any of these measures of educator effectiveness and if so, for whom (e.g. teacher, principal, new hires, teacher leaders, effective teachers):
Overall evaluation score
Formal observation rating
Student growth data
Informal information (and if so, what)
Other?
Ask the purpose of reviewing, analyzing or using the measure. For example:
Was it used to help choose this as a high-priority activity?
Was it used to identify effective educators?
Was it used to identify schools or individuals to target?
Was it used to determine compensation amounts, like bonuses or stipends?
Was it used to identify topics for professional development or how to provide targeted professional development?
Were the measures revised to support this activity using TSL funds?
If “no”, move to next set of questions. If “yes” ask: How was the measure revised or improved? For example, for each measure, ask:
Overall evaluation score - e.g. data that contributed to score, how items were weighted
Formal observation rating - e.g. number of observations, training of observers, rubric used.
Student growth data - e.g. revised model such as a value-added model or other growth model, revised data used, such as replacing state assessment scores with SLOs.
Informal information not used for evaluation purposes (and if so, what) – e.g. walkthroughs and rubric used, who observes, number of observations, how data is recorded.
Check information in Table C.3 to help track respondent’s answers and where to follow up.
Table C.3 – Educator effectiveness checklist for interviewer only
Measures of educator effectiveness used (A) |
Data used for teacher, principal or other (check if “yes”) (B) |
How measure was used was explained (C) |
Notes about how measure was used |
Were TSL funds used to revise the measure to support this activity? (D) |
a. Overall evaluation score Yes No |
Teacher Principal other (explain) |
Yes No |
|
Yes No |
b. Formal observation rating Yes No |
Teacher Principal other (explain) |
Yes No |
|
Yes No |
c. Student growth data Yes No |
Teacher Principal other (explain) |
Yes No |
|
Yes No |
d. Informal information Yes No |
Teacher Principal other (explain) |
Yes No |
|
Yes No |
e. Other (describe): Yes No |
Teacher Principal other (explain) |
Yes No |
|
Yes No |
C1.4. Whether and how data was used.
This next set of questions focuses on learning more about whether and how you may have used or improved your use of data other than educator effectiveness data, such as data on student attendance or behavior, student characteristics, or teacher attendance, to either choose to focus on this activity or to support the activity in some way.
Did you analyze, review, or use data to help you choose to focus on this activity or to support the program? If “No”, go to next section.
Please explain (1) what data you are using (or used), (2) how you are using them, (3) if improving measures based on the data was part of this effort (and if so, how they were revised), and (4) if the TSL grant funds were used at all to help revise or improve the measures.
Follow-up questions if not mentioned when answering the questions above:
What measure did you use?(see table C4 for list of measures)
Who was the measure on (students, teachers, principals)?
How did you use the measure to help identify the focus of the activity? (e.g. to determine who will receive the support, who to partner with, which opportunities to provide, etc.)
Do you monitor how the activity is carried out? If so, how often do you refer to data when monitoring the activity? What data do you use?
What data is used to measure outcomes?
Anything else?
Check information in Table C.4 to help track respondent’s answers and where to follow up.
Table C.4 – Other data (not educator effectiveness measures) checklist for interviewer only
Measure (A) |
Measure for student, teacher, principal or other (check if “yes”) (B) |
How measure was used was explained (C) |
Notes about measure and how it was used |
Were TSL funds used to revise the measure to support this activity? (D) |
a. Attendance Yes No |
Student Teacher Principal other (explain) |
Yes No |
|
Yes No |
b. Demographic characteristics Yes No |
Student Teacher Principal other (explain) |
Yes No |
|
Yes No |
c. Behavioral Yes No |
Student Teacher Principal other (explain) |
Yes No |
|
Yes No |
d. Retention Yes No |
Student Teacher Principal other (explain) |
Yes No |
|
Yes No |
e. Other (describe): Yes No |
Student Teacher Principal other (explain) |
Yes No |
|
Yes No |
C1.5. Improvements to data systems.
Sometimes in order to support a new or revised activity, it can require changes to the district’s data system itself. In order to conduct or support [activity] (reference the actual activity you are discussing, e.g. providing incentives to attract minority or bilingual teachers), did you change or improve your data systems?
For example: did you (check all that they say “yes” to and probe on them):
Convert paper records to electronic records stored in system? [Probe on what kind of data was converted and why.]
Incorporate new or additional data into your human capital management system? [Probe on what kind of data, student, teacher, principals, was incorporated and why.]
Revise your system so you can link different types of data? [Probe on what data were linked and why.]
Create data management strategies or tools? [Probe on what strategies or tools were created and why, such as a data dictionary, data manual, or data use protocols.]
Improve how people can access data? [Probe on who got access to the data and why.]
Create or update software? [Probe on what - such as an online application process – and how the process works and why they set it up that way.]
Anything else? [Probe on how the process works and why they set it up that way.]
If “yes” that they revised their data systems to support the activity ask:
Did you use TSL funds to support this effort?
C1.6. Successes and challenges.
Finally, please describe what you consider your successes in terms of this activity or challenges that you have faced? [Probe: If they mention any challenges, ask if they have been able to address those challenges or have plans to address them, and if so, how.]
C2. Support improvement of teachers or school leaders
One of the activities you mentioned on our last call was [insert activity] [Interviewer briefly summarize any explanation about the activity that the respondent provided on the last call.] Is my summary accurate? If yes: Great, I’d like to ask you a few questions about the activity. If no: Thanks for noting that. Can you please describe the activity?
C2.1. New/Revised activity.
Ask questions in Table C1 for the relevant activity (e.g. C2a). Note that the respondent should answer whether an activity is new under the TSL grant. If the activity was new under another grant (i.e. TIF), then we consider that activity to be an extension or revision. If you are not sure if it is new vs extended/revised, make notes about what the district was doing previously as well as the activity funded by the TSL grant and ask project leadership.
Table C2. Whether activities under “support improvement of teachers or school leaders” are new or revised/extended
Activity |
Is this a new activity, or an extension or revision of an existing activity? |
a. improve or develop new teacher induction activities |
New activity Extension/ revision |
b. provide general professional development for teachers or school leaders |
New activity Extension/ revision |
c. provide additional tools or resources focused on instructional improvement to teachers or school leaders |
New activity Extension/ revision |
d. provide targeted professional development based on teacher or school needs |
New activity Extension/ revision |
e. provide additional support for teachers (e.g. providing additional support from a colleague or creating or hiring new mentor positions) |
New activity Extension/ revision |
f. If “Other” fits here, (describe): |
New activity Extension/ revision |
C2.2. Specific questions about the activity.
For the high priority activity, first ask generally about the program. Probe on the specific follow-up questions under the relevant activity (e.g. C2c), as appropriate.
Please tell me generally about the activity. (Interviewer can use information collected from first call to prompt the respondent.)
Can you tell me why this activity is a high-priority activity for your TSL grant? (General probes for any activity: Why did your district choose this particular activity to focus on? What is the goal? How do you expect this program will help you achieve this goal?)
Potential follow-up questions if not mentioned when answering the general questions above:
Who provides the support? Is it from within the district or an outside provider?
What topics are focused on? How is this determined?
What is the frequency and number of sessions or activities educators or schools receive?
Which schools receive the program? How were they identified?
Which individuals are targeted for the support? How were they identified? [If not mentioned, ask if novice teachers (and how they define that) or struggling teachers were targeted.]
C2.3. Use of educator effectiveness data to accomplish the activity.
This next set of questions focuses on learning more about whether and how you may have used educators’ effectiveness measures to either choose to focus on this activity or to support the activity in some way. We will also ask whether you have improved or revised the effectiveness measures for this activity. A separate set of questions we ask later will cover other types of data (like data on student attendance or behavior).
First ask generally about their use of educator effectiveness data:
Did you analyze, review, or use educator effectiveness data to help you choose to focus on this activity or to support the activity in any way? If “No”, go to Question C1.4.
Please explain (1) what educator effectiveness data you are using (or used), (2) how you are using them, (3) if improving the measures was part of this effort (and if so, how they were revised), and (4) if the TSL grant funds were used at all to help revise or improve the measures.
Follow-up probes if not mentioned when answering earlier questions:
Ask if used any of these measures of educator effectiveness and if so, for whom (e.g. teacher, principal, new hires, teacher leaders, effective teachers):
Overall evaluation score
Formal observation rating
Student growth data
Informal information (and if so, what)
Other?
Ask the purpose of reviewing, analyzing or using the measure. For example:
Was it used to help choose this as a high-priority activity?
Was it used to identify effective educators?
Was it used to identify schools or individuals to target?
Was it used to determine compensation amounts, like bonuses or stipends?
Was it used to identify topics for professional development or how to provide targeted professional development?
Were the measures revised to support this activity using TSL funds?
If “no”, move to next set of questions. If “yes” ask: How was the measure revised or improved? For example, for each measure, ask:
Overall evaluation score - e.g. data that contributed to score, how items were weighted
Formal observation rating - e.g. number of observations, training of observers, rubric used.
Student growth data - e.g. revised model such as a value-added model or other growth model, revised data used, such as replacing state assessment scores with SLOs.
Informal information not used for evaluation purposes (and if so, what) – e.g. walkthroughs and rubric used, who observes, number of observations, how data is recorded.
Check information in Table C.3 to help track respondent’s answers and where to follow up.
Table C.3 – Educator effectiveness checklist for interviewer only
Measures of educator effectiveness used (A) |
Data used for teacher, principal or other (check if “yes”) (B) |
How measure was used was explained (C) |
Notes about how measure was used |
Were TSL funds used to revise the measure to support this activity? (D) |
a. Overall evaluation score Yes No |
Teacher Principal other (explain) |
Yes No |
|
Yes No |
b. Formal observation rating Yes No |
Teacher Principal other (explain) |
Yes No |
|
Yes No |
c. Student growth data Yes No |
Teacher Principal other (explain) |
Yes No |
|
Yes No |
d. Informal information Yes No |
Teacher Principal other (explain) |
Yes No |
|
Yes No |
e. Other (describe): Yes No |
Teacher Principal other (explain) |
Yes No |
|
Yes No |
C2.4. Whether and how other data was used.
This next set of questions focuses on learning more about whether and how you may have used or improved your use of data other than educator effectiveness data, such as data on student attendance or behavior, student characteristics, or teacher attendance, to either choose to focus on this activity or to support the activity in some way.
Did you analyze, review, or use data to help you choose to focus on this activity or to support the program? If “No”, go to next section.
Please explain (1) what data you are using (or used), (2) how you are using them, (3) if improving measures based on the data was part of this effort (and if so, how they were revised), and (4) if the TSL grant funds were used at all to help revise or improve the measures.
Follow-up questions if not mentioned when answering the questions above:
What measure did you use?(see table C4 for list of measures)
Who was the measure on (students, teachers, principals)?
How did you use the measure to help identify the focus of the activity? (e.g. to determine who will receive the support, who to partner with, which opportunities to provide, etc.)
Do you monitor how the activity is carried out? If so, how often do you refer to data when monitoring the activity? What data do you use?
What data is used to measure outcomes?
Anything else?
Check information in Table C.4 to help track respondent’s answers and where to follow up.
Table C.4 – Other data (not educator effectiveness measures) checklist for interviewer only
Measure (A) |
Measure for student, teacher, principal or other (check if “yes”) (B) |
How measure was used was explained (C) |
Notes about measure and how it was used |
Were TSL funds used to revise the measure to support this activity? (D) |
a. Attendance Yes No |
Student Teacher Principal other (explain) |
Yes No |
|
Yes No |
b. Demographic characteristics Yes No |
Student Teacher Principal other (explain) |
Yes No |
|
Yes No |
c. Behavioral Yes No |
Student Teacher Principal other (explain) |
Yes No |
|
Yes No |
d. Retention Yes No |
Student Teacher Principal other (explain) |
Yes No |
|
Yes No |
e. Other (describe): Yes No |
Student Teacher Principal other (explain) |
Yes No |
|
Yes No |
C2.5. Improvements to data systems.
Sometimes in order to support a new or revised activity, it can require changes to the district’s data system itself. In order to conduct or support [activity] (reference the actual activity you are discussing, e.g. providing incentives to attract minority or bilingual teachers), did you change or improve your data systems?
For example: did you (check all that they say “yes” to and probe on them):
Convert paper records to electronic records stored in system? [Probe on what kind of data was converted and why.]
Incorporate new or additional data into your human capital management system? [Probe on what kind of data, student, teacher, principals, was incorporated and why.]
Revise your system so you can link different types of data? [Probe on what data were linked and why.]
Create data management strategies or tools? [Probe on what strategies or tools were created and why, such as a data dictionary, data manual, or data use protocols.]
Improve how people can access data? [Probe on who got access to the data and why.]
Create or update software? [Probe on what - such as an online application process – and how the process works and why they set it up that way.]
Anything else? [Probe on how the process works and why they set it up that way.]
If “yes” that they revised their data systems to support the activity ask:
Did you use TSL funds to support this effort?
C2.6. Successes and challenges.
Finally, please describe what you consider your successes in terms of this activity or challenges that you have faced? [Probe: If they mention any challenges, ask if they have been able to address those challenges or have plans to address them, and if so, how.]
C3. Retain and reward high-performing educators
One of the activities you mentioned on our last call was [insert activity] [Interviewer briefly summarize any explanation about the activity that the respondent provided on the last call.] Is my summary accurate? If yes: Great, I’d like to ask you a few questions about the activity. If no: Thanks for noting that. Can you please describe the activity?
C3.1. New/Revised activity.
Ask questions in Table C3 for the relevant activity (e.g. C3a). Note that the respondent should answer whether an activity is new under the TSL grant. If the activity was new under another grant (i.e. TIF), then we consider that activity to be an extension or revision. If you are not sure if it is new vs extended/revised, make notes about what the district was doing previously as well as the activity funded by the TSL grant and ask project leadership.
Table C3. Whether activities under “retain and reward high-performing educators” are new or revised/extended
Activity |
Is this a new activity, or an extension or revision of an existing activity? |
a. provide additional pay to teachers or school leaders based on their performance, and without requiring extra work (e.g. performance bonuses, for earning National Board Certification, or salary increases based on performance) |
New activity Extension/ revision |
b. provide additional pay to teachers or school leaders for taking on extra duties (e.g. stipend for being a teacher-leader) |
New activity Extension/ revision |
c. provide additional pay for high-performing teachers or school leaders to transfer to high-need schools |
New activity Extension/ revision |
d. provide additional pay for high-performing teachers to teach hard-to-staff subjects |
New activity Extension/ revision |
e. provide additional opportunities or career pathways for highly effective teachers or school leaders, without additional pay |
New activity Extension/ revision |
f. If “Other” fits here, (describe): |
New activity Extension/ revision |
C3.2. Specific questions about the activity.
For the high priority activity, first ask generally about the program. Probe on the specific follow-up questions under the relevant activity, as appropriate.
Please tell me generally about the activity. (Interviewer can use information collected from first call to prompt the respondent.)
Can you tell me why this activity is a high-priority activity for your TSL grant? (General probes for any activity: Why did your district choose this particular activity to focus on? What is the goal? How do you expect this program will help you achieve this goal?)
Follow-up questions if not mentioned when answering the general questions above for any of the categories (C3a – C3e):
Is the program for teachers or principals?
How did you determine who was eligible to participate?
How did you choose who was chosen (for the program, incentive, etc)?
Which schools are implementing the program or are eligible? How were they identified?
If appropriate: How much do teachers/school leaders receive? Does the compensation (bonuses, stipends, salary increases) vary? If so, how?
If appropriate: Did teacher/principal receive additional training to support this activity?
C3.3. Use of educator effectiveness data to accomplish the activity.
This next set of questions focuses on learning more about whether and how you may have used educators’ effectiveness measures to either choose to focus on this activity or to support the activity in some way. We will also ask whether you have improved or revised the effectiveness measures for this activity. A separate set of questions we ask later will cover other types of data (like data on student attendance or behavior).
First ask generally about their use of educator effectiveness data:
Did you analyze, review, or use educator effectiveness data to help you choose to focus on this activity or to support the activity in any way? If “No”, go to Question C1.4.
Please explain (1) what educator effectiveness data you are using (or used), (2) how you are using them, (3) if improving the measures was part of this effort (and if so, how they were revised), and (4) if the TSL grant funds were used at all to help revise or improve the measures.
Follow-up probes if not mentioned when answering earlier questions:
Ask if used any of these measures of educator effectiveness and if so, for whom (e.g. teacher, principal, new hires, teacher leaders, effective teachers):
Overall evaluation score
Formal observation rating
Student growth data
Informal information (and if so, what)
Other?
Ask the purpose of reviewing, analyzing or using the measure. For example:
Was it used to help choose this as a high-priority activity?
Was it used to identify effective educators?
Was it used to identify schools or individuals to target?
Was it used to determine compensation amounts, like bonuses or stipends?
Was it used to identify topics for professional development or how to provide targeted professional development?
Were the measures revised to support this activity using TSL funds?
If “no”, move to next set of questions. If “yes” ask: How was the measure revised or improved? For example, for each measure, ask:
Overall evaluation score - e.g. data that contributed to score, how items were weighted
Formal observation rating - e.g. number of observations, training of observers, rubric used.
Student growth data - e.g. revised model such as a value-added model or other growth model, revised data used, such as replacing state assessment scores with SLOs.
Informal information not used for evaluation purposes (and if so, what) – e.g. walkthroughs and rubric used, who observes, number of observations, how data is recorded.
Check information in Table C.3 to help track respondent’s answers and where to follow up.
Table C.3 – Educator effectiveness checklist for interviewer only
Measures of educator effectiveness used (A) |
Data used for teacher, principal or other (check if “yes”) (B) |
How measure was used was explained (C) |
Notes about how measure was used |
Were TSL funds used to revise the measure to support this activity? (D) |
a. Overall evaluation score Yes No |
Teacher Principal other (explain) |
Yes No |
|
Yes No |
b. Formal observation rating Yes No |
Teacher Principal other (explain) |
Yes No |
|
Yes No |
c. Student growth data Yes No |
Teacher Principal other (explain) |
Yes No |
|
Yes No |
d. Informal information Yes No |
Teacher Principal other (explain) |
Yes No |
|
Yes No |
e. Other (describe): Yes No |
Teacher Principal other (explain) |
Yes No |
|
Yes No |
C3.4. Whether and how data was used.
This next set of questions focuses on learning more about whether and how you may have used or improved your use of data other than educator effectiveness data, such as data on student attendance or behavior, student characteristics, or teacher attendance, to either choose to focus on this activity or to support the activity in some way.
Did you analyze, review, or use data to help you choose to focus on this activity or to support the program? If “No”, go to next section.
Please explain (1) what data you are using (or used), (2) how you are using them, (3) if improving measures based on the data was part of this effort (and if so, how they were revised), and (4) if the TSL grant funds were used at all to help revise or improve the measures.
Follow-up questions if not mentioned when answering the questions above:
What measure did you use?(see table C4 for list of measures)
Who was the measure on (students, teachers, principals)?
How did you use the measure to help identify the focus of the activity? (e.g. to determine who will receive the support, who to partner with, which opportunities to provide, etc.)
Do you monitor how the activity is carried out? If so, how often do you refer to data when monitoring the activity? What data do you use?
What data is used to measure outcomes?
Anything else?
Check information in Table C.4 to help track respondent’s answers and where to follow up.
Table C.4 – Other data (not educator effectiveness measures) checklist for interviewer only
Measure (A) |
Measure for student, teacher, principal or other (check if “yes”) (B) |
How measure was used was explained (C) |
Notes about measure and how it was used |
Were TSL funds used to revise the measure to support this activity? (D) |
a. Attendance Yes No |
Student Teacher Principal other (explain) |
Yes No |
|
Yes No |
b. Demographic characteristics Yes No |
Student Teacher Principal other (explain) |
Yes No |
|
Yes No |
c. Behavioral Yes No |
Student Teacher Principal other (explain) |
Yes No |
|
Yes No |
d. Retention Yes No |
Student Teacher Principal other (explain) |
Yes No |
|
Yes No |
e. Other (describe): Yes No |
Student Teacher Principal other (explain) |
Yes No |
|
Yes No |
C3.5. Improvements to data systems.
Sometimes in order to support a new or revised activity, it can require changes to the district’s data system itself. In order to conduct or support [activity] (reference the actual activity you are discussing, e.g. providing incentives to attract minority or bilingual teachers), did you change or improve your data systems?
For example: did you (check all that they say “yes” to and probe on them):
Convert paper records to electronic records stored in system? [Probe on what kind of data was converted and why.]
Incorporate new or additional data into your human capital management system? [Probe on what kind of data, student, teacher, principals, was incorporated and why.]
Revise your system so you can link different types of data? [Probe on what data were linked and why.]
Create data management strategies or tools? [Probe on what strategies or tools were created and why, such as a data dictionary, data manual, or data use protocols.]
Improve how people can access data? [Probe on who got access to the data and why.]
Create or update software? [Probe on what - such as an online application process – and how the process works and why they set it up that way.]
Anything else? [Probe on how the process works and why they set it up that way.]
If “yes” that they revised their data systems to support the activity ask:
Did you use TSL funds to support this effort?
C3.6. Successes and challenges.
Finally, please describe what you consider your successes in terms of this activity or challenges that you have faced? [Probe: If they mention any challenges, ask if they have been able to address those challenges or have plans to address them, and if so, how.]
C4. Improve or create measures to assess educator effectiveness used to inform performance-based compensation decisions
One of the activities you mentioned on our last call was [insert activity] [Interviewer briefly summarize any explanation about the activity that the respondent provided on the last call.] Is my summary accurate? If yes: Great, I’d like to ask you a few questions about the activity. If no: Thanks for noting that.
(Note that we assume we do not need to ask if this is “new” or “revised/extended” because that information is provided if they say they do the activity.)
C4.1. Who measure assesses.
For [activity], does that measure assess teachers? Principals? Anyone else?
Table C4. Whether activities under “improve measures to assess educator effectiveness used to inform performance-based compensation decisions” are for teachers, principals, or other
Activity |
Measure for teacher, principal or other (check if “yes”) |
a. Revise how to measure or calculate overall educator effectiveness |
Teacher Principal other (explain) |
b. Improve/revise teacher or principal observation measure |
Teacher Principal other (explain) |
c. Improve/revise student achievement growth measure |
Teacher Principal other (explain) |
d. Generate a new measure of educator effectiveness |
Teacher Principal other (explain) |
e. If “Other” fits here, (describe): |
Teacher Principal other (explain) |
C4.2. Specific questions about the activity.
For the high priority activity, first ask generally about the program. Probe on the specific follow-up questions under the relevant activity, as appropriate.
Please tell me generally about the activity. (Interviewer can use information collected from first call to prompt the respondent.)
Can you tell me why this activity is a high-priority activity for your TSL grant? (General probes for any activity: Why did your district choose this particular activity to focus on? What is the goal? How do you expect this program will help you achieve this goal?)
Follow-up questions if not mentioned when answering the general questions above:
Revise how to measure or calculate overall educator effectiveness
Did you revise what data contributed to the overall score?
Did you revise how measures were weighted into the overall score?
Improve/revise teacher or principal observation measure
Did you revise the number of times the teacher/principal was observed?
Did you revise how observers were trained?
Did you change the rubric used?
Improve/revise student achievement growth measure
Did you revise how you measured student achievement growth (e.g revised data that went into a model [like a value-added model] or the model itself?
Did you revise how student achievement was assessed (such as replacing state assessment scores with SLOs)?
Generate a new measure of educator effectiveness
What is this measure based on? Why did you decide to start using it?
Improve/revise informal measures of educator effectiveness
What is this measure based on (e.g. walkthroughs, review of teacher or student materials)?
Did you change how the information was evaluated (such as who observes, the number of observations, or how data is recorded)?
C4.3. Successes and challenges.
Finally, please describe what you consider your successes in terms of this activity or challenges that you have faced? [Probe: If they mention any challenges, ask if they have been able to address those challenges or have plans to address them, and if so, how.]
C5. Improve or create other measures aside from educator effectiveness measures used to inform other human capital management decisions
One of the activities you mentioned on our last call was [insert activity] [Interviewer briefly summarize any explanation about the activity that the respondent provided on the last call.] Is my summary accurate? If yes: Great, I’d like to ask you a few questions about the activity. If no: Thanks for noting that.
(Note that we assume we do not need to ask if this is “new” or “revised/extended” because that information is provided if they say they do the activity.)
C5.1. Who measure assesses.
For [activity], does that measure assess teachers? Principals? Students? Anyone else?
Table C5. Whether activities under “improve or create other measures aside from educator effectiveness measures used to inform other human capital management decisions” are for teachers, principals, or other
Activity |
Note if measure was for teachers, principals, students, or other |
a. Revise an existing measure |
Teacher Principal Student Other (explain) |
b. Create a new measure |
Teacher Principal Student Other (explain) |
c. If “Other” fits here, (describe): |
Teacher Principal Student Other (explain)) |
C5.2. Specific questions about the activity.
For the high priority activity, first ask generally about the program. Probe on the specific follow-up questions under the relevant activity, as appropriate.
Please tell me generally about the activity. (Interviewer can use information collected from first call to prompt the respondent.)
Can you tell me why this activity is a high-priority activity for your TSL grant? (General probes for any activity: Why did your district choose this particular activity to focus on? What is the goal? How do you expect this program will help you achieve this goal?)
Follow-up questions if not mentioned when answering the general questions above, as appropriate:
What data did this effort focus on? For example, student attendance, student behavior, student achievement, student characteristics, teacher turnover, teacher characteristics, teacher attendance, principal characteristics.
Did you revise how you use an existing measure? If yes, please explain.
Did you create a new measure? If so, please explain why, what it measures, and how you are using it.
Did you revise how you use an existing measure? For example, did you revise a measure to to help identify teacher candidates, effective teachers, or at-risk students?
C5.3. Successes and challenges.
Finally, please describe what you consider your successes in terms of this activity or challenges that you have faced? [Probe: If they mention any challenges, ask if they have been able to address those challenges or have plans to address them, and if so, how.]
C6. Improve data systems to make more efficient and user-friendly or incorporate new data
One of the activities you mentioned on our last call was [insert activity] [Interviewer briefly summarize any explanation about the activity that the respondent provided on the last call.] Is my summary accurate? If yes: Great, I’d like to ask you a few questions about the activity. If no: Thanks for noting that.
(Note that we assume we do not need to ask if this is “new” or “revised/extended” because that information is provided if they say they do the activity.)
C6.1. Who measure assesses is not applicable. Go to C6.2
C6.2. Specific questions about the activity.
For the high priority activity, first ask generally about the program. Probe on the specific follow-up questions under the relevant activity, as appropriate.
Please tell me generally about the activity. (Interviewer can use information collected from first call to prompt the respondent.)
Can you tell me why this activity is a high-priority activity for your TSL grant? (General probes for any activity: Why did your district choose this particular activity to focus on? What is the goal? How do you expect this program will help you achieve this goal?)
Follow-up questions if not mentioned when answering the general questions above, as appropriate:
Convert paper records to electronic records stored in system
Which data did you convert?
Incorporate new or additional data on students, teachers, or principals in the district’s human capital management system
Which data did you incorporate?
Revise system so it can link together different types of data
Which systems did you revise or link?
What, specifically, was revised?
Create data management strategies or tools?
Who created the strategy or tool? (e.g. data dictionary, data manual, or data use protocols)
How was the strategy or tool disseminated to appropriate stakeholders?
Improve data accessibility
Who did the improvement allow to access the data?
What data became more accessible? How?
Create or update software
Who has access to the software?
What is the purpose of the software?
How often do they use it? How do you monitor use?
C6.3. Successes and challenges.
Finally, please describe what you consider your successes in terms of this activity or challenges that you have faced? [Probe: If they mention any challenges, ask if they have been able to address those challenges or have plans to address them, and if so, how.]
That is all the questions I have for you today. Thank you for your time.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Kristin Hallgren |
| File Modified | 0000-00-00 |
| File Created | 2021-01-15 |