Implementation Evaluation of the ADVANCE Program

ADVANCE IMPLEMENTATION EVALUATION
Telephone Interview Protocol
Program Director
Expected duration: 1 to 1.25 hours
Note to interviewers:
There are a lot of probes here. All do not have to be used.
The probes are included to provide examples of additional questions that might be asked, as appropriate.
Use your judgment and stay within the time limit.

A. Background Information
1. What was your position within the ADVANCE program at ____________________?
How long were you involved with ADVANCE?
2. As ______________________, what was your role vis à vis the program?
3. Did you hold any other positions at your institution?
4. Who was responsible for the day-to-day operation of the program?
B. Administrative and Governance Structure
5. To whom did you report as ________________ of the ADVANCE program? (If applicable)
What was the administrative structure of the program?
6. Where within the institutional structure was your ADVANCE program located? How and why
was the decision made to locate it there?
7. Was there an advisory committee to your program? (If so, please describe its role and
composition.)
C. Program Goals and History
Before we talk about your specific activities, I am curious about how you got to the point of
developing and submitting the proposal.
8. Tell me how your effort got started. What led your institution to submit a proposal? Who was
involved in the process? (Probe: Had there been earlier activities to address gender equity in the
sciences at your institution? Did your ADVANCE program build on previous institutional
initiatives?)
9. Describe the proposal creation process. How did you come to define the problem/challenge as
you did? Who was involved ? (Probe: senior campus leaders, faculty, students, others.) What
role did they play? (Probe: active role, consultation, kept informed, etc.)
10. What was the specific gender equity “problem” that you wished to address with program
funding? Please describe how your institution identified this “problem.” (Probe: What data
were used? Who was involved? What role did they play? Why was the decision made to
address that specific “problem”?)
11. Did the definition of the “problem” or your understanding of the problem change over time?
How? What was the effect of this change on your approaches?

1

12. What was the rationale for selecting the STEM departments (mentioned in your proposal) as the
sites for intervention? (Probe: Were any STEM departments excluded? Why?)
13. What were the goals of the project? How well do the goals of the project fit into the overall
mission of the institution?
D. Program Implementation
Now let’s talk about your ADVANCE project as it was funded by NSF and implemented by your
institution.
Program Functions

Culture
Climate

14. Given the goals of the program and how you defined your problem(s), how did you decide on the
appropriate strategies for addressing these goals?
(Probe: What was the process for choosing main strategies? Who was involved? What ideas or
frameworks guided your choices? Was the choice based on a theory of change or other theory or
specific assumptions?)
15. What were the strategies and approaches proposed by your institution to address ADVANCE
goals?
16. What were the main components of the ADVANCE Program?
(Interviewer: Verify from proposal what these are and get respondent to confirm. Ask only if
components are different from the strategies and approaches described above. Make note of
components that were dropped and/or added and find out why.)
17. Why were these components chosen? How were they selected?
(Probe: Who was involved in the selection process? What was the rationale for choosing these?
Based on any research or theories? Institutional culture? Prior activities?)
18. Where did you get resistance (active and/or passive)? How did you overcome it sufficiently to
move ahead?
19. Were there other efforts or changes outside of ADVANCE that contributed to or hindered the
agenda?
20. Of the components of the program, which were the most successful? Why do you think these
components succeeded? (Probe: For reasons why these components were successfully
implemented and why they were successful in advancing project goals.)
21. Which components were the least successful? What were the reasons for their lack of success?
(Probe: Failure in implementation or were they not successful in achieving goals?) Were any of
these abandoned at some point in the five-year period (may be answered in Q16)?
22. What factors contributed the most to the successful implementation of your ADVANCE
program?
23. What factors posed the greatest challenge to program success?
24. What are some conditions that you feel have to be present in order to successfully implement an
ADVANCE-type program?
25. What role do you feel institutional or departmental culture played in facilitating or deterring the
success of the project? Please describe. (Probe: Did you encounter resistance to your project
because of the institutional or departmental culture? How did you build on institutional or
departmental culture to gain support for your project?)

2

Evidence of Institutional Transformation
Culture
Climate

Resource
s

Policies
Practices
Structure
s

26. Has the climate of the institution or the STEM departments become more gender-friendly?
(Probe: Are group interactions different? Has the language used by the institution/ departments
become more sensitive to gender issues? Do the topics that dominate campus conversations
reflect greater awareness of gender issues? Are new stakeholders participating in institutional
discourse? Have relations with key stakeholders changed to reflect more support for gender
equity?)
27. In addition to NSF funds, what resources contributed to the implementation of the ADVANCE
program? (Probe: Did the institution or department provide funding—in the form of summer
stipends, research funding, recruitment incentives, additional staff, release time, in kind
contributions such as space, equipment, etc. Was funding obtained from external sources?)
28. Has the institutional or departmental budget been reallocated in ways that support improved
gender equity for faculty in S&E? Was this due to ADVANCE?
29. Has the institution or STEM departments leveraged ADVANCE funding to attract additional
funds to promote gender equity for faculty? (Was funding obtained from external sources?)
30. Have there been changes to or establishment of policies and practices at either the departmental
or institutional levels to support gender equity among science faculty? Was this due to
ADVANCE?
31. Have new institutional or departmental structures been created to support gender equity for S&E
faculty? Was this due to ADVANCE?
32. Have decision-making structures changed to support improved gender equity for S&E faculty?
Was this due to ADVANCE? If so, how? in what ways?

E. Program Outcomes
33. Overall, how successful has your ADVANCE Program been in achieving longer-term outcomes?
(Probe: What data can you cite to support that claim?)
Have any of the components of your ADVANCE Program been institutionalized?
34. Has your program design been disseminated or replicated? If so, please describe.
35. Has your project produced any publications or conference papers? If so, may I have a list of
publications and papers?

Interviewer: FOR INSTITUTIONS THAT HAVE LEADERSHIP AWARDS FROM NSF ADVANCE
36. Did your institution receive a Leadership grant? If so, did this grant help to further the goals of
the IT grant? How?
Interviewer: FOR INSTITUTIONS THAT HAVE PAID AWARDS FROM NSF ADVANCE
37. Did your institution receive a PAID grant? If so, did this grant help to disseminate or replicate
your IT program? How?

3