Potential Principal Investigator Interview Protocol

Attachment 3f:

Potential Principal Investigator

Interview Protocol

Noyce Program Evaluation Interview Protocol: Potential PI

OBJECTIVES:

The purpose of this inquiry is to obtain insights from representatives from eligible, non-Noyce institutions and individuals who have never received a Noyce Award regarding the STEM teacher preparation/leadership field overall and how their institutions may/may not contribute to it, their perceptions about the Noyce Program's impact and effectiveness, facilitators, and barriers to applying for a Noyce Award, and any changes in the likelihood of applying over time. For those who have never applied, one of the selection criteria will be at least some familiarity with the Noyce Program.

INTERVIEWER SCRIPT:

INTRODUCTION

• Hello, my name is [NAME OF INTERVIEWER], and I will be facilitating this confidential interview today.

• I am an independent evaluation and research consultant. I work for WhitworthKee Consulting. We are partnering with the National Science Foundation to evaluate the Robert Noyce Teacher Scholarship Program and identify areas that are working well and opportunities for growth. NSF has contracted the WKC Evaluation Team as a third-party evaluator so that those inside and outside of the Noyce community can express their true perceptions and experiences with Noyce without having personal information tied to responses.

• Joining us today is my colleague [NAME OF CO-INTERVIEWER/ NOTETAKER] who will function in the background to take notes and observe the session. They will remain in the background without video but may also ask follow-up questions.

• As you may know, the interviews were organized by your role or position within the Noyce Program when possible. This is the potential PI interview.

• The purpose of this interview is to discuss your perceptions of the Noyce Program and the STEM teacher preparation/leadership field overall. There are no right or wrong answers, and please be straightforward with your responses. You may not have experience with or know the answer to every question. Please feel free to share with us if you don’t know the answer to the question or it doesn’t apply to you. This interview will last no more than 60 minutes.

• You can skip any question that you do not want to answer, and you can stop participating in this interview at any time. Your participation in this interview will not impact you or your institution’s ability to receive funding from NSF either now or in the future.

CONFIDENTIALITY

Your answers in today’s interview will remain private. The WKC Evaluation Team will not connect your name or any personal details to your responses. The results will be summarized to reflect the overall group’s perspectives. Before sharing the information with external parties (e.g., NSF, NSF’s website, reports to Congress, Noyce events, other conferences, publication audiences, or other professional communities), the team will make sure that any identifying details are removed.

We would like to record this interview session so that we can ensure that we have time to think deeply about your responses. To maintain data security, the recording will be stored in a password-protected cloud-based drive. It will not be shared with anyone except the researchers working on the Noyce Program evaluation. Please verbally acknowledge your consent to recording by saying, “I agree.” [INTERVIEWER NOTE: MAKE NOTE OF PARTICIPANT CONSENT (YES/NO) IN THE TRACKER]

Do you have any questions before we get started?

[IF YES, PLEASE ANSWER ANY QUESTIONS AS BEST YOU CAN. IF YOU ARE UNABLE TO ANSWER A QUESTION, PLEASE REACH OUT TO THE PROJECT LEAD OR CO-LEAD]

[IF NO, BEGIN THE INTERVIEW QUESTIONS]

INTERVIEW QUESTIONS

[INTERVIEWER NOTE: START RECORDING IF PARTICIPANT CONSENTED; IF THEY DID NOT, BOTH THE INTERVIEWER AND NOTETAKER WILL TAKE NOTES TO CAPTURE AS MUCH DATA AS POSSIBLE] I’m going to start the recording now.

Background

[INFORMATION ON NAME, CURRENT ROLE, AND INSTITUTION WILL BE CAPTURED FROM THE DEMOGRAPHIC QUESTIONNAIRE – SHARE THE RESPONDENT’S DEMOGRAPHIC QUESTIONNAIRE ANSWERS AND VERIFY THEY ARE CORRECT.

IF ANY INFORMATION IS INCORRECT, PLEASE NOTE THE RESPONDENT’S CORRECTED INFORMATION AND GIVE THE UPDATED INFORMATION TO THE PROJECT LEAD OR CO-LEAD]

1. We’d first like to start our questions off by asking: What has been your previous experience supporting training the next generation of STEM teachers? Can you please describe for us a bit how you have been involved and what that has looked like? [BACKGROUND]

STEM Teacher Preparation Field

INTERVIEWER SCRIPT: In the next section, we will ask questions about your views on funding the STEM teacher preparation/leadership field overall, as well as your views on the Noyce Program funding.

2. From your perspective, what specific parts of the STEM teacher preparation field do you think should be emphasized and funded? [1A]

INTERVIEWER SCRIPT: As you may know, the Noyce Program provides funding to higher education institutions via (a) scholarships, (b) stipends, and (c) other means of programmatic support to recruit and prepare science, technology, engineering, and mathematics (STEM) majors and other professionals to become K-12 teachers and teacher leaders.

3. What is your overall perception of the Noyce Program? [3A]

4. If the Noyce Program could fund anything (the sky is the limit), what would you suggest that NSF fund to recruit, prepare, and retain STEM teachers and teacher leaders in high-need school districts? [1]

1. Probe: Can you identify specific areas within STEM education that are currently underserved or overlooked that could benefit from additional funding? [1A]

2. Probe: Are there specific populations of individuals in STEM education that are currently underserved or overlooked that you think could benefit from additional funding?

3. Probe: How could the Noyce Program be expanded to further support the professional development of STEM teachers and teacher leaders? [1A]

INTERVIEWER SCRIPT: The Noyce Program addresses the critical need for recruiting, preparing, and retaining highly effective elementary and secondary mathematics and science teachers and teacher leaders who persist as classroom teachers in high-need school districts.

[INTERVIEWER NOTE: READ DEFINITION VERBATIM AND PROVIDE TEXT IN CHAT OR ON PAPER.] Per the Higher Education Act of 1965, a high-need school district is defined as having at least one school that meets at least one of the following criteria:

1. not less than 20% of the children served by the agency are from low-income families;

2. serves at least 10,000 children from low-income families;

3. is eligible for funding under the Small, Rural School Achievement Program under 20 U.S.C. 7345(b); or

4. is eligible for funding under the Rural and Low-Income School Program under 20 U.S.C. 7351(b); and,

Meets one of the following criteria:

1. has a high percentage of teachers not teaching in the academic subject areas or grade levels in which the teachers were trained to teach; or

2. has a high teacher turnover rate or a high percentage of teachers with emergency, provisional, or temporary certification or licensure.

5. In this definition, what are your thoughts on the way the term "high-need school district" is described? Do you think the definition supports the Noyce Program’s goals of reaching high-need K-12 students? If yes, why? If not, why not? [2A]

1. Probe [IF PARTICIPANT(S) INDICATE(S) THE DEFINITION SHOULD BE DIFFERENT]: What alternative definition would you recommend to further help the Noyce Program reach high-need K-12 students? [2A]

2. Probe: What are your perspectives on how Noyce scholars and fellows can teach in a school that is not high-need (but may reside in a district that is high-need by definition) to fulfill their program requirements?

6. How do you or would you determine if a school is high-need with respect to Noyce's definition? Are there certain data or metrics you use or would use?

   1. Probe: Would it be helpful if Noyce’s definition was more specific? Less specific? Why or why not?

7. Have you ever considered applying for an award as part of the Noyce Program? If so, which one(s)? [1B, 1C, 3, 3B, 3C, 3D]

[INTERVIEWER NOTE: IF YES TO Q7, CONTINUE ON TO Q8; IF NO, SKIP TO Q11]

8. Have you ever been involved in applying for a Noyce Award or one of its capacity-building grants? If yes, please describe how you were involved, how many times, the approximate year that was, and the Track if you recall. [BACKGROUND] [INTERVIEWER NOTE: IF NEEDED – REFER TO THE INFORMATION ABOUT THE FOUR TRACKS AND OTHER AREAS BELOW]

1. Track 1: Scholarships and Stipends

2. Track 2: Teaching Fellowship

3. Track 3: Master Teaching Fellowship

4. Track 4: Noyce Research

5. Capacity Building

6. Research Experiences in STEM Settings

9. Are there any factors that would make it easy for your institution to apply for Noyce funding? [IF NEEDED: FOR EXAMPLE, TO WHAT EXTENT ARE TRAINING, INSTITUTIONAL SUPPORT, MATERIALS AND RESOURCES, EASE OF APPLICATION, ETC., FACTORS WHEN APPLYING FOR NOYCE FUNDING?] [1B]

10. Are there any factors that would make it challenging to apply for Noyce funding? [IF NEEDED: FOR EXAMPLE, TO WHAT EXTENT ARE COST-SHARING REQUIREMENTS, DEGREE OF ALIGNMENT OF THE TRAINING MODELS BEING USED BY INSTITUTIONS WITH THE NOYCE REQUIREMENTS, OVERALL FUNDING LIMITATIONS, FUNDING LIMITATIONS FOR STUDENTS, WAYS IN WHICH INFORMATION IS COMMUNICATED, ETC., BARRIERS TO APPLYING?] [1C]

1. Probe: How does the burden of applying for the Noyce Award compare to other similar funding opportunities in STEM education, if you are aware of other similar opportunities? [IF NEEDED: FOR EXAMPLE, IS IT EASIER, MORE DIFFICULT, ABOUT THE SAME?] [1B, 1C, 3, 3B, 3C, 3D]

2. Probe: Was there sufficient technical support available from NSF when you had questions or needed assistance with your application? For example, was there sufficient technical support for submitting the proposal via Research.gov? [1B, 3, 3C, 3D]

3. Probe: What types of support or resources do you think would be most beneficial for PIs to apply for a Noyce Award (e.g., technical assistance, funding)? [1B, 3, 3C, 3D]

11. Earlier in our conversation, you indicated that you [APPLIED/DID NOT APPLY] for a Noyce grant:

    1. [IF RESPONDENT APPLIED]: What motivated you to apply? What factors or which people influenced your decision? For example, what factors existed within society, the larger field, at the institution level, and/or at the college, department, and faculty level?

    2. [IF RESPONDENT DID NOT APPLY]: What factors influenced your decision to not apply? For example, have you not heard of it, was it too burdensome to apply, etc.? [1B, 2A]

    3. [IF RESPONDENT DID NOT APPLY]: Probe: What are some specific challenges or complexities that that have prevented you or others from your institution from applying? For example, what challenges or complexities within your state, the larger field, at the institution level, and/or at the college, department, and faculty level? [1B, 1C, 3, 3B, 3C, 3D]

12. What do you think could be learned from the Noyce Program and applied to other similar STEM teacher preparation and teacher leader initiatives? [1D]

Changes to the Solicitation

INTERVIEWER SCRIPT: This next section of questions depends on if you have been familiar with the Noyce solicitation over the years.

13. How familiar are you with the Noyce solicitation? [BACKGROUND] [INTERVIEWER NOTE: CAN USE A SCALE FROM 1-10; IF FAMILIARITY of 5+, PROCEED TO Q13. IF BELOW 5, PROCEED TO Q20]

14. During the time you’ve been familiar with the Noyce Program solicitation, how have substantial changes to the solicitation that NSF posts impacted how you perceive the Program? [3]

    1. Probe: What specific changes to the solicitation have impacted your perception? For example, how have changes in participant support allocation, elimination of cost-sharing requirements, etc., impacted how you think about the Program? [3]

15. What ways are you aware of that the recruitment process has changed? [3A]

    1. Probe: In what ways did these changes impact whether you applied for Noyce funding or not? [3B]

16. In what ways has the application process changed? [3A]

    1. Probe: To what extent has the amount of effort it takes to complete a Noyce Award application changed over time? [3A]

    2. Probe: In what ways did these changes impact whether you applied for Noyce funding or not? [3B]

Merit Review Process and Program Outcomes

[INTERVIEWER NOTE: IF YES to Q8, ASK QUESTIONS 17-19; IF NO TO Q8, SKIP TO Q20]

INTERVIEWER SCRIPT: In the next few questions, we will discuss your views on the merit review and funding process for the Award.

17. Have you gone through the merit review process? [IF PARTICIPANT SAYS NO, CONTINUE TO QUESTION #18; IF YES, CONTINUE WITH PROBES BELOW]

    1. Probe: What are your impressions/perspectives on the Noyce application merit review process? [4, 4B]

    2. Probe: Do you think that the merit review process is conducted fairly and objectively? Can you share any specific experiences or observations that shaped your view on this? [4, 4B]

    3. Probe: What are your thoughts on the expertise of the reviewers involved in the merit review process? Did they seem adequately knowledgeable about the field and the goals of the Program? [4, 4B]

    4. Probe: What were your perceptions of the feedback you received on your Noyce solicitation application? [4B]

    5. Probe: Overall, how satisfied are you with the merit review process, and what changes, if any, would you recommend? [4, 4B]

18. What were your perceptions of the feedback you received during the negotiation process? [4B]

    1. Probe: From your experience, how does the merit review process of the Noyce Program affect which projects get funded? [4, 4B]

19. Have you been through a Noyce declination process? [IF PARTICIPANT SAYS YES, MOVE TO PROBE A.; IF NO, SKIP TO QUESTION #20]

    1. Probe: If you have been through a Noyce Award declination process, what were your impressions of that process? [4B]

    2. Probe: How did NSF describe the reasons for declination to you? Did you review any comments you received from the Program Officer in Research.gov? [4B]

[INTERVIEWER NOTE: ALL RESPONDENTS RECEIVE LAST QUESTION]

Wrap Up

20. As we wrap up our time together, would you like to share any lessons learned during your involvement with the Noyce Program or any growth opportunities that could help improve the Program? [5]

INTERVIEWER SCRIPT: Thank you for your time and for sharing your experiences and perceptions. This will be extremely helpful for NSF as they identify areas in Noyce that are working well and opportunities for growth. As I mentioned at the beginning of our interview, all information that you provided will remain confidential, and any identifying details will be removed before any information is shared with external parties. Finally, all of the recordings will only be made accessible to the WKC Evaluation Team.