Evaluation of the Comprehensive Technical Assistance Centers

OUTCOMES-FOCUSED INTERVIEW GUIDE FOR CENTER STAFF
NATIONAL EVALUATION OF COMPREHENSIVE CENTERS
The OMB Control Number for this information collection is and the expiration date is .
According to the Paperwork Reduction Act of 1995, persons are not required to respond to this collection of
information unless it displays a currently valid OMB control number and expiration date. Responding to this
interview is voluntary. Public reporting burden for this collection of information is estimated to average 90
minutes response. Send comments regarding this burden estimate or any other aspect of this collection of
information to [email protected].

Introduction
Suggested introductory remarks (Cover all main points here):
My name is ___________. I work for IMPAQ International, and we are conducting a National
Evaluation of the Comprehensive Technical Assistance Centers Program. The U.S. Department
of Education’s Institute for Education Sciences contracted with us to conduct this study.
As part of our evaluation, we are interviewing directors, managers, and staff at the Centers to
understand the work that you do. The purpose of the evaluation is to gather data to describe the
work of the Centers and how the Centers build the capacities of state departments of education,
and to report that information to the Department of Education. What you have to say is
important to us and we appreciate your helping us understand your work. We want to assure you
that participation in this interview is voluntary. We estimate that this interview will take 90
minutes.
This is the last of three rounds of site visits and interviews conducted for this evaluation.
Interviews this year will focus on the outcomes of your work with states and other constituents.
Read the following to the respondent(s):
“Information collected for this study comes under the confidentiality and data protection
requirements of the Institute of Education Sciences (The Education Sciences Reform Act of
2002, Title I, Part E, Section 183). Responses to this data collection will be used only for
research purposes. Findings from the interview data will be reported in summary form and
individuals will not be identified by name. Specific Centers may be identifiable when occasional
examples of their work are provided. Other than this situation that we have made respondents
aware of, we will not provide information that identifies you to anyone outside the study team,
except as required by law.”

IMPAQ International, LLC
Outcomes-Focused Interview Guide for Center Staff

1

Before we begin, do you have any questions about the purpose of the evaluation or our
confidentiality policy? Do you mind if we audio record our session for the accuracy of our note
taking?
As we proceed with the interview, we will first discuss a specific project in this priority area
(Great Teachers/Leaders or Early Learning), following up on a project we have discussed
previously. We will then ask a few questions about your work overall in this priority area (Great
Teachers/Leaders or Early Learning).
(Site liaisons will have communicated when scheduling the visit that two one-hour interviews
will be administered, one for each of the two priority areas.)
In this interview, we are interested in focusing on outcomes. Please keep in mind that the
purpose of this evaluation is not to monitor performance, but to describe and understand
capacity-building processes and outcomes.
Throughout the interview, please take care to explicitly identify the different actors and their
different roles. We need to be able to clearly distinguish actions of the Center, SEAs, and
partner organizations and to identify specific outcomes for specific constituents. It’s easy for
these to blur together.

Updated Description of Specific Project
(Site liaisons will have communicated with the Centers before the interview to identify the
project to be discussed. Site liaisons will prepare for the visit by reviewing the project
information that they will be asked to confirm.)
We would now like to discuss the __________ project, which we have discussed in previous
visits.
We want to first quickly confirm the project features and understand any new developments in
the project since our last discussion.
1. We understand that the name of this project is ________. Is this the name by which the
Center refers to and reports on the project? (If not, please provide the correct or updated
name.)
2. Based on our last interview in spring 2016, we understand that the goals of the project are
______, including the capacity-building goals of __________. Is this correct? Have there been
any modifications to those goals or the objectives of the project since our interview in spring
2016? If so, please describe these modifications and what led to them. Have there been any
other important changes in the context of the project?
IMPAQ International, LLC
Outcomes-Focused Interview Guide for Center Staff

2

3. We understand that the start date for this project was ____________. Is this correct? What
was the end date (or what is the anticipated end date)?
4. We understand that the organizational constituents for this project are (specific SEAs, LEAs,
IHEs, other agencies or some combination, broad national constituency) ________________.
Is this correct? If not, please provide the current organizational constituents served by the
project. Have any of the constituents changed since our last interview in 2016?
5. We understand that other organizations or partners involved in providing technical
assistance on this project include _____. Is this correct? Have there been any changes since
our last interview in spring 2016?
6. The major project activities that we discussed in our last interview in spring 2016 included
__________. Were there any changes in activities, or new developments in the
implementation of the project, since our last interview in spring 2016 that we should be
aware of? Please explain. (If the project is not yet completed) Please describe any planned
activities for the remainder of the project.

Measuring Outcomes
7. We‘d like to ensure that we understand how you measure and track outcomes.
a. How do/did you track, record, or measure changes in the SEAs’ capacity over time for
this project? How do you assess whether the Center’s activities are helping to drive
these changes? Do you do this differently for other project outcomes (other than
capacity-building)?
b. What instruments/tools, if any, do you use?
c. What were the challenges involved in measuring outcomes for this project? How did the
Center address these challenges?
d. Are the measures used for this project typical of what you use as a Center for all
projects? If there are major differences, please describe.

Outcomes of Project
8. We’d now like to discuss the project outcomes, and the TA strategies that produced each of
the outcomes.
We are now going to read through a list of types of SEA capacity and ask you to look at the
handout for reference. Please tell us which of these categories of capacity has been
IMPAQ International, LLC
Outcomes-Focused Interview Guide for Center Staff

3

produced by this project so far, and we will then follow up about the details. We will give
you a chance later to describe other kinds of outcomes you achieved from this project. We
understand that not all types of capacity are relevant for all projects.
(Interviewer now reads through the handout, including types of capacity and definitions. The
respondents will have received it in advance. After the reading of the handout, the
respondent is asked to identify the categories of outcomes produced by this project.
For each outcome identified by the respondent, probe for a detailed description of capacity
changes, and ensure that each of the following is addressed:)
i. What is the agency or agency staff doing differently now that they did not do before?
(Interviewer, probe for detail on specific behaviors, skills, or processes as relevant.)
ii. Was this an intended or unintended outcome?
iii. How do you know that your work led to this change? (If the respondent has difficulty
answering, refer to the measures discussed in question #7 and ask if they were used.)
iv. What technical assistance strategy(ies) or services worked well in producing this
change? How and why? What strategies or services did not work as well? Why?
v. What were the challenges in producing this outcome? How did the Center address
these challenges?
The handout will include these:
a. Knowledge and skills or human capacity: Any changes in SEA staff knowledge, expertise
or skills resulting from the project.
o Includes increased content knowledge or policy knowledge.
o Management or technical skills.
b. Organizational capacity, including structural, fiscal, or material capacity: Any changes in
SEA organizational capacity resulting from the project. By changes in organizational
capacity, we mean any of the following:
o SEA agency restructuring.
o Changes in SEA agency processes and procedures.
o Changes in communication and coordination across staff and divisions of the SEA
or other agencies.
o Improvements in SEA use of resources.
o Changes in SEA performance management and continuous improvement.
o Other.

IMPAQ International, LLC
Outcomes-Focused Interview Guide for Center Staff

4

c. Policy development or design: Any changes in the SEAs’ capacity to develop or inform
the development and design of state policy, including recommendations, advising to
Boards of Education and legislators.
d. Capacity for policy implementation, including political capacity: Any changes in how
SEAs implement policy. What we mean here are changes in how SEAs do any of the
following:
o How SEAs work to “roll out,” disseminate, or spread a policy or practice
throughout the state.
o How SEAs work with LEAs and other stakeholders in the state, for example, in
supporting, guiding, or communicating with them.
e. Other capacity changes for the SEA that you would like to describe.
f. Capacity changes for other constituents, such as local education agencies or institutions
of higher education.

9. Has the project had any other outcomes that we haven’t discussed, including any unrelated
to capacity building? If so, please describe.
(Interviewer, elicit a detailed description and ensure that each of the following is addressed.)
a. What is happening in the agency or state now that did not happen before? (Interviewer,
probe for detail on specific behaviors, skills, policies or processes as relevant.)
b. Was this an intended or unintended outcome?
c. How do you know that your work led to this change? (If the respondent has difficulty
answering, refer to the measures discussed in question #7 and ask if they were used.)
d. What technical assistance strategy(ies) or services worked well in producing this
change? What strategies did not work as well? Why?
e. What were the challenges in producing this outcome? How did the Center address these
challenges?
10. (If not already addressed) Have the constituents begun to take over or own the project
activities, information, or processes to any degree? Please explain how they are doing this
and sustaining the work. Do you know what their next steps are for moving forward? Do
you think they will require additional support to sustain the work, and if so, in what way?
11. (For completed projects) Did the project achieve its goals? Were there any goals or project
objectives that the Center was not able to achieve? If so, to what do you attribute this?

IMPAQ International, LLC
Outcomes-Focused Interview Guide for Center Staff

5

(For projects still in process) Is the project meeting its goals so far? Are there any goals or
project objectives that the Center has not been able, or may not be able, to achieve? If so,
to what do you attribute this?

Priority Area Overview
We will now ask a few questions about this overall priority area (Great Teachers/Leaders or
Early Learning).
12. Since the inception of the Center grant, has the context or focus for this priority area (Great
Teachers/Leaders or Early Learning, as per the specific interview) changed for your
constituents, beyond what we’ve already discussed? (If so) How has it changed? How has
this affected the work of your Center?
If the above project was the only one in this priority area, you may skip questions 13-15 and
move to question 16.
13. Considering all of the projects within this priority area over the last 5 years, to what extent
and in what ways did your Center contribute to your constituents' capacity, beyond what
we have already discussed? What were the major outcomes that you were able to achieve
in this area?
14. What strategies were the most effective in this priority area and why? What “works” for
building capacity in this priority area? What did not “work?”
15. What challenges (if any outside of those already discussed) made it difficult to build needed
capacity in this priority area? How did you address the challenges? Please give examples.
How likely are these challenges (or others) to persist moving forward? What additional
supports are needed to continue building or sustaining capacity in this area?
16. Did the connections between strategies and outcomes that we have discussed today align
with your theory of action as a Center? If not, why not? Have you changed or modified
your theory of action (the underlying principles that inform your approach to building
capacity) to take these results into account? Please explain.
17. Have you learned anything new about how to build SEA capacity over the course of this
grant? What lessons have you learned doing this work?
Thank you very much for your time today. Your input has been very helpful.

IMPAQ International, LLC
Outcomes-Focused Interview Guide for Center Staff

6