Evaluation of the Comprehensive Technical Assistance Centers

OUTCOMES-FOCUSED INTERVIEW GUIDE FOR
TECHNICAL ASSISTANCE RECIPIENTS
NATIONAL EVALUATION OF COMPREHENSIVE CENTERS

The OMB Control Number for this information collection is and the expiration date is .
According to the Paperwork Reduction Act of 1995, persons are not required to respond to this collection of
information unless it displays a currently valid OMB control number and expiration date. Responding to this
interview is voluntary. Public reporting burden for this collection of information is estimated to average one
hour per response. Send comments regarding this burden estimate or any other aspect of this collection of
information to [email protected].

Introduction
Suggested introductory remarks (Cover all main points here):
My name is and I work for IMPAQ International. We are researching the
Comprehensive Technical Assistance Centers. The U.S. Department of Education’s Institute for
Education Sciences contracted with us to conduct this study.
(If applicable) You may recall that we talked last year about [name of Center].
As part of our study, we will be interviewing recipients of technical assistance from the
Comprehensive Centers. The purpose of the study is to describe the work of the Centers,
especially how the Centers build the capacities of state departments of education, and to help
inform the future work of the Centers. What you have to say is important to us. We want to
assure you that participation in this interview is voluntary. We estimate that this interview will
take 60 minutes.
Read the following to the respondent(s):
“Information collected for this study comes under the confidentiality and data protection
requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 2002,
Title I, Part E, Section 183). Responses to this data collection will be used only for research
purposes. Findings from the interview data will be reported in summary form and individuals
will not be identified by name. However, respondents’ roles and the CC they work with may be
identified in the report, which may lead to individuals’ being identified. Other than this situation
that we have made respondents aware of, we will not provide information that identifies you to
anyone outside the study team, except as required by law.”

IMPAQ International, LLC
Outcomes-Focused Interview Guide for Center Staff

1

Before we begin, do you have any questions about the purpose of the evaluation or our
confidentiality policy?
To start, do you mind if we audio record our session for the accuracy of our note taking?
(Interviewer: Respondents will have been identified by the Centers as technical assistance
recipients of projects in the two priority areas —teacher/leader effectiveness and early learning.
Respondents should be staff of state agencies or offices responsible for administration of one of
the two priority areas. In most cases the respondent will work for an office or division within the
state department of education. Occasionally the respondent may work for a state agency other
than the department of education, such as an office of early childhood. When administering the
protocol, you may substitute for “state education agency” or “SEA” or “your agency or office”
the specific name of the respondent’s office or agency as she or he generally refers to it.)

Individual and Project Information
1. We understand that you work for (name of agency and office or division if relevant). Is this
correct? Please describe your position and your responsibilities.
2. When did you first start working with the [Center name]?
3. We’ll be discussing the [project name] that you worked on with the [Center name]. We
understand that this project (Interviewer provide description based on recent Center
interview.) Is this an accurate description, including an accurate description of the Center’s
role in the project? (If not, ask the respondent to explain or add information.) If the project
is ongoing, please describe any planned activities for the remainder of the project.
4. Please describe your involvement in this project since it started. Were others in your
agency involved? If so, please describe their involvement.
5. We understand that the goals of the project for your agency are _____________? Is this
correct? If not, please correct this or describe any additional goals.
6. Have there been changes in your needs, goals or objectives for the project since its
inception? If so, how has this affected your work with the Center on this project?

Measuring Outcomes
7. We’d like to understand how you measure(d) project progress and outcomes. How do you
measure or determine whether the project is meeting its goals and objectives? Has the
Center worked with you on measuring or tracking outcomes for the project? If so, please

IMPAQ International, LLC
Outcomes-Focused Interview Guide for Center Staff

2

describe how they have done this. Do you need more help in measuring project progress
and outcomes?

Outcomes of the Project
8. We’d now like to discuss the project outcomes, especially the Center’s role in producing
them.
We are now going to read through a list of types of agency capacity and ask you to look at
the handout for reference. Please tell us which of these categories of capacity has been
built by the Center’s work with you on this project so far, and we will then follow up about
the details. We will give you a chance to describe other kinds of project outcomes as well.
Interviewer now reads through the handout, including types of capacity and definitions. After
the reading of the handout, the respondent is asked to identify which outcomes on the list
were produced by this project.
For each outcome identified by the respondent, probe for a detailed description of capacity
changes, and ensure that each of the following is addressed:
i.

What are you, your staff, or your agency doing differently now that you did not do
before? (Interviewer, probe for detail on specific behaviors, skills, or processes as
relevant.)

ii.

Was this an intended or unintended outcome of this project (from your agency’s
perspective)?

iii.

How do you know that the Center’s work on this project contributed to this change?
What evidence do you have? (Interviewer, refer to answers under #8 if applicable.)

iv.

What technical assistance strategies or services provided by the Center worked well in
producing this change? How and why? What strategies or services did not work as
well? Why?

v.

What are the challenges in producing this outcome (building this type of capacity
change or other outcome in your agency)? How did the Center address these
challenges?

The handout will include these:
a. Knowledge and skills or human capacity: Any changes in SEA staff knowledge, expertise
or skills resulting from the project.
IMPAQ International, LLC
Outcomes-Focused Interview Guide for Center Staff

3

o Includes increased content knowledge or policy knowledge.
o Management or technical skills.
b. Organizational capacity, including structural, fiscal, or material capacity: Any changes in
SEA organizational capacity resulting from the project. By changes in organizational
capacity, we mean any of the following:
o SEA agency restructuring.
o Changes in SEA agency processes and procedures.
o Changes in communication and coordination across staff and divisions of the SEA
or other agencies.
o Improvements in SEA use of resources.
o Changes in SEA performance management and continuous improvement.
o Other.
c. Policy development or design: Any changes in the SEAs’ capacity to develop or inform
the development and design of state policy, including recommendations, advising to
Boards of Education and legislators.
d. Capacity for policy implementation, including political capacity: Any changes in how
SEAs implement policy. What we mean here are changes in how SEAs do any of the
following:
o How SEAs work to “roll out,” disseminate, or spread a policy or practice
throughout the state.
o How SEAs work with LEAs and other stakeholders in the state, for example, in
supporting, guiding, or communicating with them.
e. Other capacity changes for the SEA that you would like to describe.

9. Has the project had any other outcomes we haven’t discussed, including any unrelated to
capacity building? If so, please describe. (Interviewer, for each change described, ask the
following:)
i. What is happening in your agency or state now that did not happen before? (Interviewer,
probe for detail on specific behaviors, skills, policies or processes as relevant.)
ii. Was this an intended or unintended outcome of this project (from your agency’s
perspective)? Did you expect the project to produce these changes, or were these
changes unexpected?

IMPAQ International, LLC
Outcomes-Focused Interview Guide for Center Staff

4

iii. How do you know that the work on Center’s work on the project contributed to this
change? What evidence do you have? (Interviewer, refer to answers under #8 if
applicable.)
iv. What activities, services or products technical assistance strategies or services provided
by the Center worked particularly well in producing this change? How and why? Did any
strategies or services did not work as well? Why?
v. What were the challenges in working on this area of capacity? What are the challenges in
producing this outcome (building this type of capacity change or other outcome in your
agency)? How did the Center address these challenges?

10. (If not already addressed) Have you begun to take over or own the project activities,
information, or processes, without the Centers’ help, to any degree? Please explain how
you are doing this and sustaining the work. Do you know what your next steps are for
moving forward? What do you think will be the biggest challenges? Do you think you will
require additional support to sustain the work, and if so, in what way?
11. (For completed projects) Did the project achieve its goals for your agency? Did the Center
meet your expectations for their work on the project? Please explain. If any goals or
objectives, or expectations for the Center, were not achieved, to what do you attribute this?
12. (For projects still in process) Is the project meeting its goals so far? Is the Center meeting
your expectations for their work on the project so far? If any goals or objectives (or
expectations for the Center) have not been achieved, to what do you attribute this?

Priority Area Overview
We’d now like to ask about your work with the Center in the area of [teacher/leader
effectiveness or early learning] overall.
13. Since 2012, has the local or state context or focus for this priority area (teacher/leader
effectiveness or early learning, as per the specific interview) changed for your agency? If
yes, how has it changed? How has this affected your work with the Center?
14. Do you believe the Centers have been effective in addressing the needs of the SEA and the
state in this priority area? If yes, please describe how and what evidence you have of their
effectiveness. If no, please explain why and how/in what ways they have not been effective
in serving the SEA/state needs.

IMPAQ International, LLC
Outcomes-Focused Interview Guide for Center Staff

5

15. Considering all of your work with the Center within this priority area over the last
[applicable number] years, to what extent and in what ways did the Center contribute to
your agency’s capacity? What were the major outcomes that achieved in this area?
16. What Center technical assistance strategies, services, or products were the most effective in
this priority area and why? What “worked” for building your agency’s capacity in this
priority area? What did not work?
17. What challenges (outside of those already discussed) made it difficult to build your agency’s
needed capacity in this priority area? Please give examples. How likely are these challenges
(or others) to persist moving forward? What additional supports might you need to
continue building or sustaining capacity in this area?
18. Do you have any lessons or recommendations for the Comprehensive Centers based on
your experience in this project or the priority area overall? Did you learn any lessons for
your own agency? Please explain.

Thank you very much for your time today. Your input has been very helpful.

IMPAQ International, LLC
Outcomes-Focused Interview Guide for Center Staff

6