Teacher interviews and focus groups

National Study of Education Data Systems and Decision Making

Teacher Interview Protocol

Introduction

The following is a suggested introduction. When you begin the interview, make these points, but use your own words for a more personal introduction:

The National Study of Educational Data Systems and Decision Making is a large-scale project sponsored by the US Department of Education to look at the ways that states, districts and schools are using data in their educational decision making. The project looks at the prevalence and quality of both the data systems that are in use nationwide and the supports in place for educational decision makers to use them. In particular, we are interested to learn about how teachers are using data from these systems to make decisions about their instruction, and what types of professional development and support teachers are receiving to learn about the systems and their applications.

We want you to know that this project does not evaluate you or your school; we’re looking to learn from your experiences to get a better understanding of how teachers and schools across the country use data systems for teaching and learning.

Before we start, I need to have you review and sign this consent form.

AFTER THEY REVIEW CONSENT FORM: Do you have any questions before we get started?

Background

1. Please tell me a little bit about your role here at this school.

1. What subjects and grade levels do you teach?

2. Do you have any additional roles at the school, especially with respect to curriculum or the use of data?

Data Use for Instructional Decision Making

The purpose of this section is to learn more about your use of this school’s data system [insert name]

2. How often do you access the system [insert name]?

3. What situations have prompted you to use the system? NOTE: If the teacher has access to a computer during the interview, ask them to open the data system and show you the reports and types of data they typically access for any situations they say led them to use data.

   1. Have you used data from the system in planning your curriculum for the year?

[IF YES] What did you conclude from the data? How often have you accessed data for this purpose?

[IF NO] Why not?

b. Have you used data from the system in forming groups of students within you class?

[IF YES] Can you describe your process for doing this? How often have you accessed data for this purpose?

2. Have you used data from the system to develop individualized learning plans for some or all of your students?

[IF YES] Can you describe your process for doing this? How often have you accessed data for this purpose this year?

3. Have you used data to make decisions about how long to spend on specific instructional objectives?

[IF YES] Can you describe your process for doing this? How often have you accessed data for this purpose this year?

4. Have you used data collected during the school year to see whether your students are on track to meet grade-level standards?

[IF YES] Can you describe your process for doing this? How often have you accessed data for this purpose?

5. Have you used data from other sources or other data systems for any of these purposes?

[IF YES] Please describe and explain why you used this other source instead of (or in addition to) the [name] data system.

4. Do you typically look at student data for these purposes on your own or as part of a group? [IF AS PART OF A GROUP] Please describe.

   1. Are different types of data looked at in groups vs. individually? Why these differences?

   2. Which approach is most useful?

[IF THE TEACHER ALREADY PROVIDED SPECIFIC EXAMPLES IN RESPONSE ABOVE QUESTIONS SKIP TO QUESTION 7. IF NO SPECIFIC EXAMPLES HAVE BEEN PROVIDED, ASK QUESTION 6]

5. Think of a specific situation in which you used data to make a decision about your instructional practice. We’d like to hear about your decision and your thinking about data’s impact on your instruction.

   1. Who did the decision concern?

   2. What data did you use and what did it tell you?

   3. What was the result of the decision?

   4. How comfortable did you feel making decisions based on the data?

6. Overall, what do you feel has been the impact of your data use on students’ educational experiences?

7. Have you had questions or needed help using the system or interpreting data from the system?

[IF YES] Did someone help you? If yes, please describe. Who can you turn to for help in your school or district?

8. How could [data system name] be more useful to you as a teacher?

   1. What other kinds of student information would you like to be able to access?

   2. In what ways could the system be easier to use?

Student Data Scenario Discussion

In this final section of the interview we will be asking teachers to look at and interpret mock-up scenarios based on hypothetical student data. These mock-ups will provide an opportunity to assess teachers’ understanding of common data interpretation tasks, and to understand what types of decisions, if any, they might make on the basis of their interpretations.

The scenarios will be developed by TWG members with data system expertise and assessment experts at a workshop to be held in Menlo Park during the summer of 2006. The scenario will ask the teacher to look at graphs or tables of student data and describe what the data tell them and the instructional inferences they would draw. Data concepts to be tested may include:

• Variance

• Measurement error

• The difference between group and individual performance

• Estimating reliability related to the number of items

We will be developing these scenarios in consultation with TWG members and experts in data-driven decision making. The timeline for creation of the scenarios is….

D-4