Project Evaluator Protocol

Project Evaluator Interview Protocol

CPATH Project Title:

Grantee Institution(s):

Interviewee Name/Institution:

Interview Date:

Interviewer(s):

Introductory Remarks:

My name is ______ and I work for SRI International, a non-profit research firm that is responsible for the evaluation of the NSF CPATH Program. The purpose of this interview is to learn more about your CPATH project, how it is being implemented, the organizations and stakeholders involved, and the effect that it has been having on various groups and organizations. Throughout our discussion, please feel free to interrupt me to ask any questions you may have or include information that you believe we should know.

Before we start, I need to have you review and sign this consent form. SRI International’s Institutional Review Board (IRB) as well as the U.S. government’s Office of Management and Budget has approved this project. Your signature on this form indicates that your participation in this interview is voluntary and that you understand that we will never directly identify or quote you based on what you tell us today.

After they review consent form: Do you have any questions before we get started?

Introduction and Background

1. Please tell me about yourself. Probe for:

a. Describe your role and responsibilities as evaluator of the CPATH project.

b. How long have you served as the evaluator?

2. How did you get involved in the CPATH project? Probe for the following:

1. How did you come to be the evaluator?

2. How were you involved in the preparation of the proposal?

Project Strategies

3. What are the key goals and strategies of this project? How does your evaluation plan seek to evaluate this project?

4. How has the evaluation effort got thus far? Probe for the following:

1. How would you describe the process of getting buy-in from the PI on the evaluation design and implementation effort?

2. How has the PI incorporated findings from the evaluation to inform project implementation?

3. Do you have a recent evaluation report you could share with us?

4. What are the key findings from the project evaluation that stand out to you?

5. Who do you think are the primary beneficiaries of your project? Note: Beneficiaries of this project may include students, faculty members, administrators, institutions of higher education, K-12 school districts, professional associations, government and non-profit organizations, and businesses and industry.
   Probe using the following:

1. Which groups of students are you targeting for participation?

2. Are you targeting any underrepresented groups for participation? If yes, please explain. (Probe for traditionally underrepresented groups including minorities, females, disabled and non-traditional computer majors.)

If project is at multiple sites, probe for information on each participating institution using the probes provided with questions in this section.

6. How has project implementation gone thus far? Probe using the following, asking respondent to provide reasons for their statements:

1. What have been the highlights or successes?

2. What misfires has the project experienced? Please describe.

3. What have been the challenges to the implementation of the project? Probe about the relevant actors – students, faculty, administrators, partners, etc.

4. What lessons have been learned from addressing these challenges?

7. What are some of the factors that have supported project implementation? Probe using the following:

1. Culture of committed faculty involvement and participation

2. Strong institutional support from department head/deans/administrators

3. Innovative curricular and pedagogical strategies

4. Student demand for course changes

8. How has the institution leveraged funding from other sources to provide additional support to the project? Probe using the following:

1. What are the specific funding sources, if any (e.g., state, federal, private)?

2. What other ongoing initiatives are focused on computing reform?

3. What other projects, if any, do you envision emerging from this project? Please explain.

Outputs

9. Overall, what influence has the CPATH project had on students and faculty? What is the evidence of these impacts? Please describe any influences as well as supporting evidence. Probe using the following asking respondents to describe the influences as well as to provide supporting evidence:

1. To what extent has the CPATH project influenced student engagement in computing and how? Please describe.

2. How has the project influenced student enrollment in computing courses? Have there been any changes in the demographics or population of students enrolled in computing courses?

3. How has the CPATH project prepared students for STEM and other careers?

4. How has the project influenced faculty members? Has it led to changes in faculty culture?

5. Have faculty members published any articles related to computing in peer-reviewed journals? (Note: this may or may not be related to the CPATH project)

10. What institutional changes would you attribute to the CPATH project? What is the evidence of this impact? Please describe any institutional changes as well as supporting evidence. Probe using the following asking respondents to describe the changes as well as supporting evidence:

1. Has the CPATH project integrated computing into other disciplines?

2. Has the CPATH project influenced the rewards/incentive structure of your institution? If yes, please explain.

3. Are the changes formal or informal?

4. In your view, who or what is driving these changes?

11. Do you think the CPATH project has created a model that could be used at other institutions? If yes, please describe the model and how you are supporting its implementation at other institutions. Probe using the following:

1. In your view, what features of the model that can be replicated easily?

2. Are there special materials or guidance needed for this model?

12. How will this project be sustained beyond the life of the CPATH grant? Probe using the following:

1. How will the CPATH project be institutionalized?

2. What other resources or supports would have to be in place to sustain this CPATH project?

3. What are the challenges to the sustainability of the CPATH project?

Conclusion

13. Is there anything else you would like to add that might help us get a better understanding of the impact the CPATH program is having on computing education at your institution (and others with which you may be involved)?

Thank you for your time.