Principal Investigator Interview Protocol

Principal Investigator Interview Protocol¹

CPATH Project Title:

Grantee Institution(s):

Interviewee Name/Title/Contact Information:

Interview Date:

Interviewer(s):

Introductory Remarks:

My name is ______ and I work for SRI International, a non-profit research firm that is responsible for the evaluation of the NSF CPATH Program. The purpose of this interview is to learn more about your CPATH project, how it is being implemented, the organizations and stakeholders involved, and the effect that it has been having on various groups and organizations. Beneficiaries of this project may include students, faculty members, administrators, institutions of higher education, K-12 school districts, professional associations, government and non-profit organizations, and businesses and industry. Throughout our discussion, please feel free to interrupt me to ask any questions you may have or include information that you believe we should know.

Before we start, I need to have you review and sign this consent form. SRI International’s Institutional Review Board (IRB) as well as the U.S. government’s Office of Management and Budget has approved this project. Your signature on this form indicates that your participation in this interview is voluntary and that you understand that we will never directly identify or quote you based on what you tell us today.

After they review consent form: Do you have any questions before we get started?

Introduction and Background

1. Please tell me about yourself. Probe for:

1. What roles do you have at this institution?

2. How long have you taught/worked at this institution?

3. What other positions have you previously held?

4. Describe your role and responsibilities related to the CPATH project.

2. Prior to the CPATH project, were you engaged in undergraduate reform activities at your institution? If so, please describe those activities.

1. What do you view as pressing issues in improving undergraduate education? (Probe using the following: enrollment and graduation trends for STEM and other disciplines, new demands on education to align more closely to industry or workforce needs, concepts related to computational thinking in the different disciplines)

3. How did the CPATH project come about? Probe for the following:

1. What was the motivation for initiating the project? How does the project align with other initiatives at your institution? Who was involved in shaping the project?

2. How did you come to be the PI?

Project Strategies

4. (1a, 3d) Please describe the nature of the teaching/learning environment for computing you are trying to create through the project. What is it designed to do at your institution (and other partner sites, if applicable)? Probe using the following:

1. How might this environment look different from traditional settings for computing?

2. How will you know you’ve been successful in creating this environment?

3. What are the core strategies you’re using to create this environment?

5. (1c) What are the core computing concepts that the CPATH project focuses on? Probe using the following:

1. How has participating in the CPATH program helped you identify and refine these concepts?

2. Define the concept of computational thinking.

6. (2a, 2b) How has your project integrated these core concepts into courses outside of traditional computing disciplines (i.e. biology, chemistry, mathematics, etc.)? Probe using the following:

1. What disciplines are involved?

2. If applicable, how has this integration played out at other participating sites/institutions?

3. What have been the challenges the project has faced in efforts to integrate these core concepts outside of traditional computing disciplines?

7. (1f, 1g, 3a) Who do you think are the primary beneficiaries of your project? Probe using the following:

1. Which groups of students are you targeting for participation?

2. Are you targeting any underrepresented groups for participation? If yes, please explain. (Probe for traditionally underrepresented groups including minorities, females, disabled and non-traditional computer majors.)

Implementation Factors

If project is at multiple sites, probe for information on each participating institution using the probes provided with questions in this section.

8. (1b, 2b) How has project implementation gone thus far? Probe using the following:

1. What have been the highlights or successes?

2. What failures has the project experienced? Please describe.

3. What have been the challenges to the implementation of the project? Probe about the relevant actors – students, faculty, administrators, partners, etc.

4. What lessons have been learned from addressing these challenges?

9. (2e, 3c) What support do faculty at your institution have to help with curriculum development or devising pedagogical strategies for teaching computational thinking? Probe using the following:

1. Describe the current incentive and awards structure at participating institutions.

2. What incentives are you offering through your CPATH project to faculty to develop curriculum, to devise pedagogical strategies and/or to participate in your project?

3. Are there any additional incentives to encourage faculty to develop innovative ways to teach computational thinking?

10. (1d, 1e, 2a) What are some of the factors that have supported project implementation? Probe using the following:

1. Culture of committed faculty involvement and participation

2. Strong institutional support from department head/deans/administrators

3. Innovative curricular and pedagogical strategies

4. Student demand for course changes

11. (3e) How has the institution leveraged funding from other sources to provide additional support to the project? Probe using the following:

1. What are the specific funding sources, if any (e.g., state, federal, private)?

2. What other ongoing initiatives are focused on computing reform?

3. What other projects, if any, do you envision emerging from this project? Please explain.

Outcomes

11. (2d, 3a) Overall, what influence has the CPATH project had on students and faculty? Please describe. Probe using the following:

1. To what extent has the CPATH project influenced student engagement in computing and how? Pleasedescribe.

2. How has the project influenced student enrollment in computing courses? Have there been any changes in the demographics or population of students enrolled in computing courses?

3. How has the CPATH project prepared students for STEM and other careers?

4. How has the project influenced faculty members? Has it led to changes in faculty culture?

5. Have faculty members published any articles related to computational thinking in peer-reviewed journals? (Note: this may or may not be related to the CPATH project)

11. (3b, 3c) Are there any institutional changes that you would attribute to the CPATH project? Please describe. Probe using the following:

1. Has the CPATH project integrated computational thinking into other disciplines?

2. Has the CPATH project influenced the rewards/incentive structure of your institution? If yes, please explain.

3. Are the changes formal or informal?

4. In your view, who or what is driving these changes?

11. (3d, 3f) Do you think the CPATH project has created a model that could be used at other institutions? If yes, please describe the model and how you are supporting its implementation at other institutions. Probe using the following:

1. Are other institutions implementing your models?

2. Are you providing materials or guidance?

3. What mechanisms are in place to sustain the relationship between you and the institutions that have adopted your model?

Community Building and Partnership Development

11. (4a, 4b, 4c, 4d) Who are the stakeholders for your project? How is information shared with this community of stakeholders? Note: Stakeholders may include other CPATH grantees, faculty within computing-related disciplines (e.g., computer science, informatics), faculty in other disciplines, university administrators, other higher education institutions, K-12 teachers, professional associations, industry and businesses, and government and non-profit organizations.

Probe using the following:

1. How inclusive is this group of stakeholders?

2. Would you say there is a shared understanding about computing competencies among stakeholders in this community?

3. How do stakeholders communicate with and learn from each other? (How is the project supporting learning and sharing of best practices around computational thinking and education amongst these stakeholders?) Probe for online sharing and communication, conferences and colloquia, informal networks and other venues for sharing best practices.

11. (5a) Are there other organizations with which your project has a significant ongoing relationship as a part of the CPATH project? Probe using the following:

1. Post secondary institutions

2. K-12 school districts

3. Government offices

4. Private companies/non-profit organizations

5. Professional membership organizations

6. Any other groups that have had or will have a significant connection to the project. Probe if there is a key person in this organization that we should talk to.

(IF YES to Q16, then ask questions 17-20; IF NO, then go to Q21)

11. (5a) To what extent has NSF funding (for the CPATH project) opened new opportunities for partnerships between multiple sectors (e.g. industry, K-12, professional associations) around computing? Please describe these new opportunities.

12. (5b, 5c) To what extent have these partnerships leveraged pre-existing relationships or new opportunities in the local science, technology and economic development environment? Probe using the following:

1. What elements outside each organization were initially perceived as needed to enhance their ability to accomplish goals?

2. How much did members believe that partnership activities would be reciprocally helpful and involve “give and take”?

3. How much did members believe that their respective stand-alone identities would be improved through a continual partnership operation and attainment of goals?

11. (5d) How much does the work done by each partner depend on the work of other members? Probe using the following:

1. To what extent is the work shared equally among the partners?

2. How do you assess the effectiveness of the partnership? What measures are you using to make that assessment?

3. How would you describe any challenges to building and maintaining effective partnerships?

11. (3f, 5e) How can multi-sector partnerships support the replication of promising models of computational thinking over the long term? What are the barriers that would inhibit sustaining these partnerships?

Conclusion

11. Is there anything else you would like to add that might help us get a better understanding of the impact the CPATH program is having on computing education at your institution (and others with which you may be involved)?

Thank you for your time.

Administrator² Interview Protocol³

CPATH Project Title:

Grantee Institution(s):

Interviewee Name/Title/Contact Information:

Interview Date:

Interviewer(s):

Introductory Remarks:

My name is ______ and I work for SRI International, a non-profit research firm, who is responsible for the evaluation of the NSF CPATH Program. The purpose of this interview is to learn more about your CPATH project, how it is being implemented, the organizations and stakeholders involved, and the effect that it has been having on various groups and organizations. Beneficiaries of this project may include students, faculty members, administrators, institutions of higher education, K-12 school districts, professional associations, government and non-profit organizations, and businesses and industry. Throughout our discussion, please feel free to interrupt me to ask any questions you may have or include information that you believe we should know.

Before we start, I need to have you review and sign this consent form. SRI International’s Institutional Review Board (IRB) has approved this project as well as the U.S. government’s Office of Management and Budget. Your signature on this form indicates that your participation in this interview is voluntary and that you understand that we will never directly identify or quote you based on what you tell us today.

(After they review consent form, ask: Do you have any questions before we get started?

Introduction and Background

1. Please tell me about yourself. Probe for:

1. What are your roles and responsibilities?

2. How long have you worked at this institution?

3. What other positions have you previously held?

4. Describe your current role related to the CPATH project.

2. What is the history of undergraduate reform at your institution? Probe using the following:

   1. What do you view as pressing issues in improving undergraduate education?

   2. What are the current undergraduate reform initiatives on this campus?

   3. How did the CPATH project come about?

3. How do reforms related to computational thinking and the CPATH project fit in with your institution’s overall vision or mission? Probe using the following:

1. How important is this project to your institution?

Project Strategies

4. (1a, 1e, 3d) Please describe the nature of the teaching/learning environment for computing you are trying to create through the CPATH project. What is it designed to do at your institution (and other partner sites, if applicable)? Probe using the following:

1. How might this environment look different from traditional settings for computing?

2. How will you know you’ve been successful in creating this environment?

3. What are the core strategies you’re using to create this environment?

5. (1f, 1g) Who do you think are the primary beneficiaries of your project? Probe using the following:

1. Which groups of students are you targeting for participation?

2. Are you targeting any underrepresented groups for participation? If yes, please explain. (Probe for traditionally underrepresented groups including minorities, females, disabled and non-traditional computer majors.)

Implementation Factors

5. (1b, 2b) From your perspective, how do you think project implementation gone thus far? Probe using the following:

1. What have been the highlights or successes?

2. What failures has the project experienced? Please describe.

3. What barriers and challenges has the project faced? What do you think still needs to be done to address these challenges

5. (2e, 3c) What support do faculty at your institution have to help with curriculum development or pedagogical strategies for teaching computational thinking? Probe using the following:

1. Describe the current incentive and rewards structure at your institution.

2. What incentives are being offered through the CPATH project to faculty to develop curriculum, to devise pedagogical strategies and/or to participate in the project?

3. Are there any additional incentives to encourage faculty to develop innovative ways to teach computational thinking?

5. (1d, 1e, 2a) What are some of the factors that have supported project implementation? Probe using the following:

1. Culture of committed faculty involvement and participation

2. Strong institutional support from department head/deans/administrators

3. Innovative curricular and pedagogical strategies

4. Student demand for course changes

5. (3e) How has the institution leveraged funding to provide additional support to the project? Probe using the following:

1. What other ongoing initiatives are focused on computing reform?

2. What other projects, if any, do you envision emerging from this project? Please explain.

Outcomes

10. (2d, 3a) Overall, what influence has the CPATH project has had on students and faculty? Probe using the following:

1. How has the project influenced student enrollment in computing courses? Have there been any changes in the demographics or population of students enrolled in computing courses?

2. How has the CPATH project prepared students for STEM and other careers?

3. How has the project influenced faculty members? Has it led to changes in faculty culture?

4. Have faculty members published any articles related to computational thinking in peer-reviewed journals? (Note: this may or may not be related to the CPATH project)

10. (3b, 3c) Are there any institutional changes that you would attribute to the CPATH project? Please describe. Probe using the following:

1. Has the CPATH project integrated computational thinking into other disciplines?

2. Has the CPATH project influenced the rewards/incentive structure of your institution? If yes, please explain.

3. How might any institutional changes be documented? For example, have there been any changes in institutional documents (course catalogs, outlines of departmental majors, etc.) as a result of integrating computational thinking across disciplines? Probe for changes in the way majors are described, description of computing courses, and the role of computing in non-computing disciplines.

10. (3d, 3f) Do you think your CPATH project has created a model that could be used at other institutions? If yes, please describe the model and how you are supporting its implementation at other institutions. Probe using the following:

1. Are other institutions implementing your models?

2. Are you providing materials or guidance?

3. What mechanisms are in place to sustain the relationship between you and the institutions that have adopted your model?

Community Building and Partnership Development

10. (4a, 4b, 4c, 4d) Who are the stakeholders for the CPATH project? How is information shared with this community of stakeholders? Note: Stakeholders may include other CPATH grantees, faculty within computing-related disciplines (e.g., computer science, informatics), faculty in other disciplines, university administrators, other higher education institutions, K-12 teachers, professional associations, industry and businesses, and government and non-profit organizations.

Probe using the following:

1. How inclusive is this group of stakeholders?

2. Would you say there is a shared understanding about computing competencies among stakeholders in this community?

3. How do stakeholders communicate with and learn from each other? (How is the project supporting learning and sharing of best practices around computational thinking and education amongst these stakeholders?) Probe for online sharing and communication, conferences and colloquia, informal networks and other venues for sharing best practices.

10. (5a) Are there other organizations with which the project has a significant ongoing relationship as a part of the CPATH project? Probe using the following:

1. Post secondary institutions

2. K-12 school districts

3. Government offices

4. Private companies/non-profit organizations

5. Professional membership organizations

6. Any other groups that have had or will have a significant connection to the project.

(IF YES to Q14, then ask questions 15-16; IF NO, then go to Q17)

15. (5b,5c) Please describe these partnerships and how they came about. Probe using the following:

1. Has NSF funding (for the CPATH project) opened new opportunities for partnerships between multiple sectors (e.g. industry, K-12, professional associations) around computing? If yes, please explain.

2. To what extent have these partnerships leveraged pre-existing relationships or new opportunities in the local science, technology and economic development environment?

15. (3f, 5e) How can multi-sector partnerships support the replication of promising models of computational thinking over the long term? What are the barriers that would inhibit sustaining these partnerships?

Conclusion

15. Is there anything else you would like to share with us about your institution or the CPATH project?

Thank you for your time.

Faculty⁴ Interview Protocol⁵

CPATH Project Title:

Grantee Institution(s):

Interviewee Name/Title/Contact Information:

Interview Date:

Interviewer(s):

Introductory Remarks:

My name is ______ and I work for SRI International, a non-profit research firm that is responsible for the evaluation of the NSF CPATH Program. The purpose of this interview is to learn more about the CPATH project at your institution, how it is being implemented, the organizations and stakeholders involved, and the effect that it has been having on various groups and organizations. Beneficiaries of this project may include students, faculty members, administrators, institutions of higher education, K-12 school districts, professional associations, government and non-profit organizations, and businesses and industry. Throughout our discussion, please feel free to interrupt me to ask any questions you may have or include information that you believe we should know.

Before we start, I need to have you review and sign this consent form. SRI International’s Institutional Review Board (IRB) has approved this project as well as the U.S. government’s Office of Management and Budget. Your signature on this form indicates that your participation in this interview is voluntary and that you understand that we will never directly identify or quote you based on what you tell us today.

After they review consent form: Do you have any questions before we get started?

Introduction and Background

1. Please tell me about yourself. Probe for:

1. What are your roles and responsibilities at this institution?

2. How long have you taught at this institution?

2. How did you first hear about the CPATH project and what motivated you to get involved? Probe using the following:

1. Direct invitation from the PI, response to departmental request, following up on personal interest in the subject, etc.

2. How have you been involved in the CPATH project? What has been your role on the project?

3. Prior to the CPATH project, were you engaged in undergraduate reform activities at your institution? If so, please describe those activities.

1. What do you view as pressing issues in improving undergraduate education? (Probe using the following: enrollment and graduation trends for STEM and other disciplines, new demands on education to align more closely to industry or workforce needs, concepts related to computational thinking in the different disciplines)

Project Strategies

4. (1a, 3d) Please describe the nature of the teaching/learning environment for computing you are trying to create in your course(s). Probe using the following:

1. How might this environment look different from traditional settings for computing?

2. How will you know you’ve been successful in creating this environment?

3. What are the core strategies you’re using to create this environment?

4. (1c) What are the core computing concepts that you focus on? Probe using the following:

1. How has the process of working with other faculty helped you identify and refine these concepts?

2. How would you define the concept of computational thinking?

4. (2a, 2b) How have you integrated these core concepts into courses in your department and into courses outside of traditional computing disciplines? Probe using the following:

1. What disciplines are involved?

2. What challenges have you face in efforts to integrate core concepts?

4. (1f, 1g, 3a) Who do you think are the primary beneficiaries of your project? Probe using the following:

1. Which groups of students are you targeting for participation?

2. Are you targeting any underrepresented groups for participation? If yes, please explain. (Probe for traditionally underrepresented groups including minorities, females, disabled and non-traditional computer majors.)

Implementation Factors

8. (1b, 2b) How has project implementation at your institution gone thus far? Probe using the following:

1. What have been the highlights or successes?

2. What failures has the project experienced? Please describe.

3. What have been the challenges to the implementation of the project? Probe about the relevant actors – students, faculty, administrators, partners, etc. What have been the lessons learned from addressing these challenges?

8. (2e, 3c) What support do faculty at your institution have to help with curriculum development or devising pedagogical strategies for teaching computational thinking? Probe using the following:

1. Describe the current incentive and awards structure at your institution.

2. What incentives are being offered through the CPATH project to faculty to develop curriculum, to devise pedagogical strategies, and/or to participate in the project?

3. Are there any additional incentives to encourage you or your peers to develop innovative ways to teach computational thinking?

8. (1d, 1e, 2a) What are some of the factors that have supported project implementation? Probe using the following:

1. Culture of committed faculty involvement and participation

2. Strong institutional support from department head/deans/administrators

3. Innovative curricular and pedagogical strategies

4. Student demand for course changes

Outcomes

Follow-up probes in this section should be tailored based on the knowledge of the interviewee.

11. (2d, 3a) Overall, what influence has the CPATH project has had on students and faculty? As appropriate, probe using the following:

1. To what extent has the CPATH project influenced student engagement in computing courses and how? Please describe.

2. How has the project influenced student enrollment in computing courses? Have there been any changes in the demographics or population of students enrolled in computing courses?

3. How has the CPATH project prepared students for STEM and other careers?

4. How has the project influenced you and other faculty members? Do you think it has led to changes in faculty culture?

5. Have faculty members published any articles related to computational thinking in peer-reviewed journals? (Note: this may or may not be related to the CPATH project)

11. (3b, 3c) Are there any institutional changes that you would attribute to the CPATH project? Please describe. If appropriate, probe using the following:

1. Has the CPATH project integrated computational thinking into other disciplines?

2. Has the CPATH project influenced the rewards/incentive structure of your institution? If yes, please explain.

3. Are the changes formal or informal?

4. In your view, who or what is driving those changes?

11. (3d, 3f) Do you think the CPATH project has created a model that could be used at other institutions? If yes, please describe the model and how the project is supporting its implementation at other institutions. If appropriate, probe using the following:

1. Are other institutions implementing your models?

2. Are you providing materials or guidance?

Community Building and Partnership Development

Questions in this section should only be asked if the interviewee is sufficiently connected to the CPATH project and the broader discussion of computing education reform within their discipline.

11. (4a, 4b, 4c, 4d) Who do you view as the stakeholders for the project? How is information shared with this community of stakeholders? Note: Stakeholders may include other CPATH grantees, faculty within computing-related disciplines (e.g., computer science, informatics), faculty in other disciplines, university administrators, other higher education institutions, K-12 teachers, professional associations, industry and businesses, and government and non-profit organizations.

As appropriate, probe using the following:

1. How inclusive is this group of stakeholders?

2. Would you say there is a shared understanding about computing competencies among stakeholders in this community?

3. How do stakeholders communicate with and learn from each other? (How is the project supporting learning and sharing of best practices around computational thinking and education amongst these stakeholders?) Probe for online sharing and communication, conferences and colloquia, informal networks and other venues for sharing best practices.

11. (5a) Do you know if there are other organizations with which the CPATH project has a significant ongoing relationship as a part of the CPATH project? Probe using the following:

1. Post secondary institutions

2. K-12 school districts

3. Government offices

4. Private companies/non-profit organizations

5. Professional membership organizations

6. Any other groups that have had or will have a significant connection to the project.

(IF YES to Q15, then ask questions 16-17; IF NO, then go to Q17)

11. (5b,5c) Please describe these partnerships and how they came about.

1. Has NSF funding (for the CPATH project) opened new opportunities for partnerships between multiple sectors (e.g. industry, K-12, professional associations) around computing? If yes, please describe.

2. To what extent have these partnerships leveraged pre-existing relationships or new opportunities in the local science, technology and economic development environment?

11. (5d) How much does the work done by each partner depend on the work of other members? As appropriate, probe using the following:

1. To what extent is the work shared equally among the partners?

2. How do you assess the effectiveness of the partnership? What measures are you using to make that assessment?

3. How would you describe any challenges to building and maintaining effective partnerships?

Conclusion

18. Is there anything else you would like to add that might help us get a better understanding of the CPATH project and computing reform activities at your institution?

Thank you for your time.

Staff⁶ Interview Protocol⁷

CPATH Project Title:

Grantee Institution(s):

Interviewee Name/Title/Contact Information:

Interview Date:

Interviewer(s):

Introductory Remarks:

My name is ______ and I work for SRI International, a non-profit research firm, who is responsible for the evaluation of the NSF CPATH Program. The purpose of this interview is to learn more about your CPATH project, how it is being implemented, the organizations and stakeholders involved, and the effect that it has been having on various groups and organizations. Beneficiaries of this project may include students, faculty members, administrators, institutions of higher education, K-12 school districts, professional associations, government and non-profit organizations, and businesses and industry. Throughout our discussion, please feel free to interrupt me to ask any questions you may have or include information that you believe we should know.

Before we start, I need to have you review and sign this consent form. SRI International’s Institutional Review Board (IRB) has approved this project as well as the U.S. government’s Office of Management and Budget. Your signature on this form indicates that your participation in this interview is voluntary and that you understand that we will never directly identify or quote you based on what you tell us today.

After they review consent form: Do you have any questions before we get started?

Introduction and Background

1. Please tell me about yourself. Probe for:

1. What’s you academic background and previous experience?

2. How long have you worked at this institution?

3. Please describe your role and responsibilities on the CPATH project.

2. How did you become involved in the CPATH project? Probe using the following:

1. Posted job opening, invitation from PI, graduate student funding opportunity, personal interest, etc.

2. Prior to the CPATH project, were you engaged in undergraduate reform activities at your institution? If so, please describe those activities.

3. In your view, what are the principal goals of the CPATH project? Probe for understanding of concept of computational thinking.

Project Strategies

4. (1a, 3d) Please describe the nature of the teaching/learning environment for computing you are trying to create through the project. What is it designed to do at your institution (and other partner sites, if applicable)? Probe using the following:

1. How might this environment look different from traditional settings for computing?

2. How will you know you’ve been successful in creating this environment?

3. What are the core strategies you’re using to create this environment?

5. (1c) What are the core computing concepts that the CPATH project focuses on? Probe using the following:

1. How has participating in the CPATH program helped you identify and refine those concepts?

2. How would you define the concept of computational thinking?

6. (2a, 2b) How has the project integrated these core concepts into courses outside of traditional computing disciplines (i.e. biology, chemistry, mathematics, etc.)? Probe using the following:

1. What disciplines are involved?

2. What have been the challenges the project has faced in efforts to integrate these core concepts outside of traditional computing disciplines?

7. (1f, 1g, 3a) Who do you think are the primary beneficiaries of the project? Probe using the following:

1. Which groups of students are you targeting for participation?

2. Are you targeting any underrepresented groups for participation? If yes, please explain. (Probe for traditionally underrepresented groups including minorities, females, disabled and non-traditional computer majors.)

Implementation Factors

If project is at multiple campuses, probe for information on each participating institution using the probes provided with questions in this section.

8. (1b, 2b) How has project implementation gone thus far? Probe using the following:

1. What have been the highlights or successes?

2. What failures has the project experienced? Please describe.

3. What have been the challenges to the implementation of the project? Probe about the relevant actors – students, faculty, administrators, partners, etc. What have been the lessons learned from addressing these challenges?

9. (1d, 1e, 2a) What are some of the factors that have supported project implementation? Probe using the following:

1. Culture of committed faculty involvement and participation

2. Strong institutional support from department head/deans/administrators

3. Innovative curricular and pedagogical strategies

4. Student demand for course changes

10. (3e) How has the project leveraged funding from other sources to provide additional support to the project? Probe using the following:

1. What are the specific funding sources, if any (e.g., state, federal, private)?

2. What other ongoing initiatives are focused on computing reform?

3. What other projects, if any, do you envision emerging from this project? Please explain.

Outcomes

11. (2d, 3a) Overall, what influence do you think the CPATH project has had on students and faculty? As appropriate, probe using the following:

1. To what extent has the CPATH project influenced student engagement in computing? Please describe.

2. How has the project influenced student enrollment in computing courses? Have there been any changes in the demographics or population of students enrolled in computing courses?

3. How has the CPATH project prepared students for STEM and other careers?

4. How has the project influenced faculty members? Has it led to changes in faculty culture?

11. (3d, 3f) Do you think the CPATH project has created a model that could be used at other institutions? If yes, please describe the model and how you are supporting its implementation at other institutions. If appropriate, probe using the following:

1. Are other institutions implementing your models?

2. Are you providing materials or guidance?

3. What mechanisms are in place to sustain the relationship between you and the institutions that have adopted your model?

Community Building and Partnership Development

11. (4a, 4b, 4c, 4d) Who are the stakeholders for the project? How is information shared with this community of stakeholders? Note: Stakeholders may include other CPATH grantees, faculty within computing-related disciplines (e.g., computer science, informatics), faculty in other disciplines, university administrators, other higher education institutions, K-12 teachers, professional associations, industry and businesses, and government and non-profit organizations.

As appropriate, probe using the following:

1. How inclusive is this group of stakeholders?

2. Would you say there is a shared understanding about computing competencies among stakeholders in this community?

3. How do stakeholders communicate with and learn from each other? (How is the project supporting learning and sharing of best practices around computational thinking and education amongst these stakeholders?) Probe for online sharing and communication, conferences and colloquia, informal networks and other venues for sharing best practices.

11. (5a) Are there organizations with which the CPATH project has a significant ongoing relationship as a part of the CPATH project? Probe using the following:

1. Post secondary institutions

2. K-12 school districts

3. Government offices

4. Private companies/non-profit organizations

5. Professional membership organizations

6. Any other groups that have had or will have a significant connection to the project. Probe if there is a key person in this organization that we should talk to.

(IF YES to Q14, then ask questions 15-16; IF NO, then go to Q17)

11. (5b,5c) Please describe these partnerships and how they came about.

1. Has NSF funding (for the CPATH project) opened new opportunities for partnerships between multiple sectors (e.g. industry, K-12, professional associations) around computing? If yes, please explain.

2. To what extent have these partnerships leveraged pre-existing relationships or new opportunities in the local science, technology and economic development environment?

11. (5d) How much does the work done by each partner depend on the work of other members? As appropriate, probe using the following:

1. To what extent is the work shared equally among the partners?

2. How do you assess the effectiveness of the partnership? What measures are you using to make that assessment?

3. How would you describe any challenges to building and maintaining effective partnerships?

Conclusion

17. Is there anything else you would like to add that might help us get a better understanding of the CPATH project?

Thank you for your time.

PARTNER⁸ INTERVIEW PROTOCOL⁹

CPATH Project Title:

Grantee Institution(s):

Interviewee Name/Title/Contact Information:

Interview Date:

Interviewer(s):

Introductory Remarks:

My name is ______ and I work for SRI International, a non-profit research firm that is responsible for the evaluation of the NSF CPATH Program. The purpose of this interview is to learn more about your CPATH project, how it is being implemented, the organizations and stakeholders involved, and the effect that it has been having on various groups and organizations. Beneficiaries of this project may include students, faculty members, administrators, institutions of higher education, K-12 school districts, professional associations, government and non-profit organizations, and businesses and industry. Throughout our discussion, please feel free to interrupt me to ask any questions you may have or include information that you believe we should know.

Before we start, I need to have you review and sign this consent form. SRI International’s Institutional Review Board (IRB) has approved this project as well as the U.S. government’s Office of Management and Budget. Your signature on this form indicates that your participation in this interview is voluntary and that you understand that we will never directly identify or quote you based on what you tell us today.

After they review consent form: Do you have any questions before we get started?

Introduction and Background

1. Please tell me about your organization. Probe for each of the following:

1. Primary mission and focus

2. Size of organization

3. Brief history and year established

4. Other major projects organization is involved in

5. Main sources of financial support

2. Please describe how you became involved with this CPATH computing education project. Probe using the following:

1. How did you first hear about it?

2. What motivated your organization to become involved?

3. What has been your organization’s role and involvement?

Project Strategies

3. (1a, 3d) Please describe the nature of the teaching/learning environments for computing that the institutions you are working with are trying to implement. What is it designed to do at the institutions? Probe using the following:

1. How might this environment look different from traditional settings for computing?

2. What are the core strategies you’re using to create this environment?

4. (1f) How will you and your partners know when you’ve been successful in creating this kind of learning environment? Probe using the following:

1. Student competencies, learning, behaviors, future career success

2. Teachers and faculty

3. Partners/Employers

5. (1c) What are the core computing competencies being addressed in the project from your perspective?

Implementation Factors

6. (2a) What steps are you and your partners taking to accomplish this vision? Probe using the following:

1. Short-term steps to implement?

2. Long-term steps to sustain and expand (i.e. keep it going)?

3. How long do you think it will take?

6. (2b) What conditions might either facilitate or hinder your ability to realize your goals? Probe using the following:

1. Institutional or organizational cultural differences/synergies

2. Institutional or organizational inertia

3. Resources (material, human, technological)

6. (3d) How is computational thinking being diffused across different disciplines and organizations through this project? What are some examples of promising models emerging from this project?

7. (4a) What are the ways you can learn about and share best practices around computing education? Probe using the following:

1. Type of communication for sharing best practices

2. Conferences and colloquia

3. Informal networks and other venues for sharing best practices.

6. (4c, 4e) To what extent do you believe different stakeholders in the field have come to agreement about the basic competencies of computational thinking? What role has NSF and the CPATH program played in promoting a shared understanding about computational thinking among different stakeholders?

7. (4f) How has the collaboration between the partners contributed to the building of strong linkages and synergies for this particular project? What is the evidence for these results?

Partnership Development

6. (5a) To what extent has the NSF funding for the CPATH project opened up opportunities for partnership between multiple sectors (e.g., industry, K-12, professional associations) around computing? Please describe these new opportunities.

7. (5b, 5c) To what extent have these partnership arrangements taken advantage of pre-existing relationships or new opportunities created by market demand or policy drivers in the environment? Probe using the following:

1. What elements outside each organization were initially perceived as needed to enhance their ability to accomplish goals?

2. How much did members believe that partnership activities would be reciprocally helpful and involve “give and take”?

3. How much did members believe that their respective stand-alone identities would be improved through a continual partnership operation and attainment of goals?

14. (5d) How much does the work done by each partner depend on the work of other members? Probe using the following:

1. To what extent do you think the work is shared equally among the partners?

2. How does the partnership assess the effectiveness of the partnership? What measures are you using to make that assessment?

3. How would you describe any challenges to building and maintaining effective partnerships?

15. (3f, 5e) How can multi-sector partnerships support the replication of promising models of computational thinking over the long term? What are the barriers that would inhibit sustaining these partnerships?

Conclusion

15. Is there anything else you would like to say about partnerships and how they can support computing education reform that was not discussed earlier?

Thank you for your time.

Student Focus Group Protocol¹⁰

CPATH Project Title:

Grantee Institution(s):

Interviewee Name/Title/Contact Information:

Interview Date:

Interviewer(s):

Introductory Remarks:

Hi, my name is [your name] and I'm here with [other site visitor], and we are researchers from SRI International, a non-profit research firm that is responsible for the evaluation of the NSF CPATH Program. The purpose of our time together is for us to learn more about computing education here, your experiences, and the effect that computing education has been having on you and other students at your institution.

We will record this conversation in case we miss something in our notes, but only [other site visitor] and I will listen to the recording. We will keep everything you say here confidential and ask you to make the same commitment—to keep what is said in this room confidential and not to repeat it to anyone else so that people can feel free to share their opinions. Throughout our discussion, please feel free to ask any questions you have or include information that you believe we should know.

Before we start, I need to have you each review and sign a consent form. The form puts in writing our guarantee that we will never directly identify or quote you based on what you tell us today. SRI International’s Institutional Review Board (IRB) has approved this project as well as the U.S. government’s Office of Management and Budget. Your signature on this form indicates that your participation in this interview is voluntary and that you understand that whatever you share with us will be kept confidential.

After they review consent form: Do you have any questions before we get started?

Introduction and Background

Let’s start by going around the room and giving your name, year in college, and your major. (For each student note the following: First Name, Gender, Year in College, Major)

Project Participation

1. (1a) What courses have you taken, or are currently enrolled in that involve computing?

2. (1a) What other non-classroom activities have you participated in, or are currently participating in that involve computing?

Student Outcomes

Students from different institutions may have varying degrees of understanding of the CPATH project being implemented at their institution. If necessary, questions should be tailored to accommodate their degree of knowledge of the CPATH project.

3. (1f) How have you benefitted from these computing courses and activities? What do you think you have gained through your participation? Probe using the following:

1. Have your experiences increased or enhanced your interest in computing? If so, how?

2. Have your courses improved your computing ability? If so, how?

3. How have your courses and experiences prepared you for your future career?

4. Have your experiences changed your career aspirations or changed your perception of the role of computing in your career? Please explain.

5. Have your experiences provided you contact with companies, faculty, or other organizations?

6. Have you obtained jobs/internships/interviews/mentoring through the CPATH project?

7. Do you think your classmates would say the same thing? Why or why not?

4. (2a) Describe a learning experience in the classroom or lab that involved computing. Note: this can be any discipline but should involve computing.

Probe using the following:

1. What was the activity, and how was computing involved?

2. What resources and tools were used?

3. What was the role of the instructor(s)/TA(s) during the activity?

4. How did you work with your peers during this activity?

5. How did this learning experience fit in with the larger unit of instruction and the overall course?

6. What did you like about this learning experience?

7. How do you hope to practically apply what you learned from this experience? (What real-world value does this learning experience have?)

5. (1g, 2a, 3a) How do your current computing courses and experiences compare to previous ones? Probe using the following:

1. What differences have you noticed between your experiences in this project and previous computing courses/experiences?

2. Have you noticed a different caliber of student in classes associated with this project compared to other computing classes you have taken? If yes, please explain.

3. Are the classes here more or less diverse in terms of females, ethnic minorities, disabled students, or non-computing majors?

6. (1c) What does the term “computational thinking” mean to you? How have your courses and experiences shaped your understanding of “computational thinking”?

7. (1f, 2a) If students have an understanding of the CPATH project at their institution: What do you think the CPATH project’s goals are for students (i.e. what do you think they hope you will get out of your participation)? How and to what extent do you think they are meeting their goals?

Future Plans

7. (3a) Do you plan to continue to be involved in computing courses and activities? Why, or why not? Probe using the following:

   1. What other courses involving computing do you plan to take?

   2. Are there other computing activities that you plan to become involved with?

9. What do you plan to do after graduation? Probe using the following:

1. Attend graduate school

2. Obtain a job using computing

3. Obtain a job in a non-computing specialty

10. How have your courses and experiences here influenced your plans after graduation?

Closing

11. Is there anything else that we didn’t ask you that you would like to share with us about your computing courses and experiences here?

Thank you for your time.

1 Throughout this interview protocol, numbers and letters in parentheses—e.g., (1a, 3d) refer to research questions and are included as a tool for evaluation analysts.

2 Administrators include department chairs, deans, provosts, or other university administrator familiar with CPATH grant activities.

3 Throughout this interview protocol, numbers and letters in parentheses—e.g., (1a, 3d) refer to research questions and are included as a tool for evaluation analysts.

4 Faculty members would include part-time or full-time faculty working with the principal investigator to implement the CPATH project.

5 Throughout this interview protocol, numbers and letters in parentheses—e.g., (1a, 3d) refer to research questions and are included as a tool for evaluation analysts.

6 Project staff could be any individual (either a part-time or full-time position) supported by the CPATH grant to assist the principal investigator with the management and administration of the project.

7 Throughout this interview protocol, numbers and letters in parentheses—e.g., (1a, 3d) refer to research questions and are included as a tool for evaluation analysts.

8 Partners are any collaborating organizations that are not institutions of higher education. These partners may be industry, K12 schools or districts, or community organizations collaborating on CPATH project activities.

9 Throughout this interview protocol, numbers and letters in parentheses—e.g., (1a, 3d) refer to research questions and are included as a tool for evaluation analysts.

10 Throughout this interview protocol, numbers and letters in parentheses—e.g., (1a, 3d) refer to research questions and are included as a tool for evaluation analysts.

A-6