AAAS Staff Focus Group Protocol

Attachment 3j:

AAAS Staff Focus Group Protocol

Noyce Program Evaluation Focus Group Protocol: AAAS Staff

OBJECTIVES:

The purpose of this inquiry is to obtain comprehensive insights from American Association for the Advancement of Science (AAAS – “triple-A-S”) staff who support the Noyce Program. The questions focus on the STEM teacher preparation/leadership field overall, their perceptions about the Noyce Program's impact and effectiveness, facilitators and barriers to applying for a Noyce Award, support needed, and any other lessons learned.

FACILITATOR SCRIPT:

INTRODUCTION

• Hello, my name is [NAME OF FACILITATOR] and I will be facilitating this confidential focus group today.

• I am an independent evaluation and research consultant. I work for WhitworthKee Consulting. We are partnering with the National Science Foundation to evaluate the Robert Noyce Teacher Scholarship Program and identify areas that are working well and opportunities for growth. NSF has contracted the WKC Evaluation Team as a third-party evaluator so that the Noyce community can express their true perceptions and experiences with Noyce without having personal information tied to responses.

• Joining us today is my colleague [NAME OF CO-FACILITATOR/ NOTETAKER] who will function in the background to take notes and observe the session. They will remain in the background without video but may also ask follow-up questions.

• As you may know, the focus groups were organized by your role or position within the Noyce Program when possible. This is the AAAS Staff focus group.

• The purpose of this focus group is to discuss your perceptions of the impact of the Noyce Program, how well the program has worked, and how the Program has changed over time. We will also discuss your perception of the factors related to the Program’s success and perceptions of those invested in the STEM education community. There are no right or wrong answers, and please be straightforward with your responses. You may not have experience with or know the answer to every question. Because roles and responsibilities can vary, please feel free to share with us if you don’t know the answer to the question or it doesn’t apply to you. This focus group will last no more than 60 minutes.

• You can skip any question that you do not want to answer, and you can stop participating in this focus group at any time. Your participation in this focus group will not impact your position at AAAS now or in the future.

CONFIDENTIALITY

Your answers in today’s focus group will remain private and will not be shared with other individuals on Noyce projects. The WKC Evaluation Team will not connect your name or any personal details to your responses. The results will be summarized to reflect the overall group’s perspectives. Before sharing the information with external parties (e.g., NSF, NSF’s website, reports to Congress, Noyce events, other conferences, publication audiences, or other professional communities), the team will make sure that any identifying details are removed.

We would like to record this focus group session so that we can ensure that we have time to think deeply about your responses. To maintain data security, the recording will be stored in a password-protected cloud-based drive. It will not be shared with anyone except the researchers working on the Noyce Program evaluation. Please verbally acknowledge your consent to recording by saying, “I agree.” [FACILITATOR NOTE: MAKE SURE TO RECEIVE A VERBAL “I AGREE” FROM ALL FOCUS GROUP PARTICIPANTS. MAKE NOTE OF PARTICIPANT CONSENT (YES/NO) IN THE TRACKER.]

Do you have any questions before we get started?

[IF YES, PLEASE ANSWER ANY QUESTIONS AS BEST YOU CAN. IF YOU ARE UNABLE TO ANSWER A QUESTION, PLEASE REACH OUT TO THE PROJECT LEAD OR CO-LEAD]

[IF NO, BEGIN THE FOCUS GROUP QUESTIONS]

FOCUS GROUP QUESTIONS

[FACILITATOR NOTE: START RECORDING IF PARTICIPANT(S) CONSENTED; IF THEY DID NOT, BOTH THE INTERVIEWER AND NOTETAKER WILL TAKE NOTES TO CAPTURE AS MUCH DATA AS POSSIBLE] I’m going to start the recording now.

FACILITATOR SCRIPT: To begin, we’d first like to hear from you a little bit about your background. [CONFIRM CURRENT ROLE]

1. How long have you been with AAAS and how long have you been in this role? [BACKGROUND]

   1. Probe: Please describe your main responsibilities related to the NSF’s Noyce Program. What activities do you specifically support and in what ways? Who do you interact with, both indirectly and directly? [IF NEEDED: FOR EXAMPLE, NOYCE PIS, POTENTIAL PIS, SCHOLARS AND FELLOWS, NSF STAFF, NOYCE PROGRAM OFFICERS, ETC.] [BACKGROUND]

2. From your perspective, what specific parts of the STEM teacher preparation field do you think should be emphasized and funded? [1A]

Noyce Program Overall

FACILITATOR SCRIPT: Thank you for those. I’m now going to transition us to discussing your thoughts are perceptions about various aspects of Noyce.

3. What is your overall perception of the Noyce Program? [2A]

   1. Probe: What is your perception of current Noyce scholars and fellows? [2B]

   2. Probe: What is your perception of former Noyce scholars and fellows? [2C]

4. In what ways, if at all, has your perception of the Noyce Program changed over time? Please describe to what this might be attributed. [2A].

[FACILITATOR NOTE: READ DEFINITION VERBATIM AND PROVIDE TEXT IN CHAT OR ON PAPER] Per the Higher Education Act of 1965, a high-need school district is defined as having at least one school that meets at least one of the following criteria:

1. not less than 20% of the children served by the agency are from low-income families;

2. serves at least 10,000 children from low-income families;

3. is eligible for funding under the Small, Rural School Achievement Program under 20 U.S.C. 7345(b); or

4. is eligible for funding under the Rural and Low-Income School Program under 20 U.S.C. 7351(b); and,

Meets one of the following criteria:

1. has a high percentage of teachers not teaching in the academic subject areas or grade levels in which the teachers were trained to teach; or

2. has a high teacher turnover rate or a high percentage of teachers with emergency, provisional, or temporary certification or licensure.

5. In this definition, what are your thoughts on the way the term "high-need school district" is described? Do you think the definition supports the Noyce Program’s goals of reaching high-need K-12 students? If yes, why? If not, why not? [2A]

   1. Probe [IF PARTICIPANT(S) INDICATE(S) THE DEFINITION SHOULD BE DIFFERENT]: What alternative definition would you recommend to further help the Noyce Program reach high-need K-12 students? [2A]

   2. Probe: What are your perspectives about how current and former Noyce scholars and fellows can teach in a school that is not high-need (but may reside in a district that is high-need by definition) to fulfill their program requirements?

6. What is your perception of the impact of the Noyce Program (i.e., after implementation)? For example, how does the Noyce Program address STEM teacher shortages and/or retaining STEM teachers in high-need schools? [2D]

   1. Probe: What factors contribute to your perception of the impact of the Program?

   2. Probe: What are the gaps in the Noyce Program's reach and impact on addressing the critical need for recruiting, preparing, and retaining highly effective STEM teachers and teacher leaders in high-need school districts.)? [2D]

7. What do you think could be learned from the Noyce Program and applied to other similar STEM teacher preparation and teacher leader initiatives? [1D]

FACILITATOR SCRIPT: In the next section, we will discuss the Noyce Scholars Grant Award application process. Please feel free to share your experiences and what you may have observed as an AAAS Staff member.

8. How would you describe the level of effort required for PIs to complete the application process for the Noyce Award? [1C]

9. What are some specific challenges or complexities that you think exist during the application phase? [1B, 1C]

10. What types of support or resources are most requested or needed for teams applying for a Noyce Award (e.g., technical assistance, funding)? Why do you think this is the case? [1B, 3, 3C, 3D]

11. We understand that the solicitation that NSF posts has changed over the years. What has contributed to some of the changes to the Noyce Program solicitations over the past 10 years? [IF NEEDED: FEEDBACK FROM SCHOOLS OR GRANTEES, FEDERAL POLICIES, INTERNAL REVIEW AND REFLECTION, FUNDING-RELATED ISSUES, STUDENT OUTCOMES, ETC.] [3, 3A, 3B]

12. What support or resources do you or your colleagues need as AAAS Staff members to better support Noyce applicants and their application to the Noyce Award or implementation of Noyce fs? [3, 3A, 3B, 3C, 3D]

13. What potential opportunities for additional collaboration might there be between NSF and AAAS? [5]

Wrap Up

14. As we wrap up our time together, would you like to share any lessons learned during your involvement with the Noyce Program or any growth opportunities that could help improve the Program? [5]

FACILITATOR SCRIPT: Thank you for your time and for sharing your experiences and perceptions. Your perspective is critical in developing a holistic understanding of the Noyce Program. As I mentioned at the beginning of our focus group, all information that you provided will remain confidential, and any identifying details will be removed before any information is shared with external parties. Finally, all of the recordings will only be made accessible to the WKC Evaluation Team.