Evaluators Focus Group Protocol

Attachment 3b:

Evaluators Focus Group Protocol

Noyce Program Evaluation Focus Group Protocol: External Evaluators

OBJECTIVES:

The purpose of this inquiry is to obtain comprehensive responses from External Evaluators concerning the measurement and evaluation of impact, effectiveness, and development of the Noyce Program over time. Additionally, it seeks to explore the factors that influence the success of projects and the perceptions of stakeholders in the STEM education community.

FACILITATOR SCRIPT:

INTRODUCTION

• Hello, my name is [NAME OF FACILITATOR], and I will be facilitating this confidential focus group today.

• I am an independent evaluation and research consultant. I work for WhitworthKee Consulting. We are partnering with the National Science Foundation to evaluate the Robert Noyce Teacher Scholarship Program and identify areas that are working well and opportunities for growth. NSF has contracted the WKC Evaluation Team as a third-party evaluator so that the Noyce community can express their true perceptions and experiences with Noyce without having personal information tied to responses.

• Joining us today is my colleague [NAME OF CO-FACILITATOR/NOTETAKER] who will function in the background to take notes and observe the session. They will remain in the background without video but may also ask follow-up questions.

• As you may know, the focus groups were organized by your role or position within the Noyce Program when possible. This is the external evaluator focus group.

• The purpose of this focus group is to discuss your perceptions of the impact of the Noyce Program overall, how well you think the program has worked, and how the program has changed over time. We will also discuss your perception of the factors related to the Project’s success and perceptions of those invested in the STEM education community. There are no right or wrong answers, and please be straightforward with your responses. You may not have experience with or know the answer to every question. Because Noyce projects and roles can vary, please feel free to share with us if you don’t know the answer to the question or it doesn’t apply to you. This focus group will last no more than 60 minutes.

• You can skip any question that you do not want to answer, and you can leave the focus group at any time. Your participation in this focus group will not impact you or your associated Noyce Program’s ability to receive funding from NSF either now or in the future.

CONFIDENTIALITY

Your answers in today’s focus group will remain private. The WKC Evaluation Team will not connect your name or any personal details to your responses. The results will be summarized to reflect the overall group’s perspectives. Before sharing the information with external parties (e.g., NSF, NSF’s website, reports to Congress, Noyce events, other conferences, publication audiences, or other professional communities), the team will make sure that any identifying details are removed.

We would like to record this focus group session so that we can ensure that we have time to think deeply about your responses. To maintain data security, the recording will be stored in a password-protected cloud-based drive. It will not be shared with anyone except the researchers working on the Noyce Program evaluation. Please verbally acknowledge your consent to recording by saying, “I agree.” [FACILITATOR NOTE: MAKE SURE TO RECEIVE A VERBAL “I AGREE” FROM ALL FOCUS GROUP PARTICIPANTS. MAKE NOTE OF PARTICIPANT CONSENT (YES/NO) IN THE TRACKER.]

Do you have any questions before we get started?

[IF YES, PLEASE ANSWER ANY QUESTIONS AS BEST YOU CAN. IF YOU ARE UNABLE TO ANSWER A QUESTION, PLEASE REACH OUT TO THE PROJECT LEAD OR CO-LEAD]

[IF NO, BEGIN THE FOCUS GROUP QUESTIONS]

FOCUS GROUP QUESTIONS

[FACILITATOR NOTE: START RECORDING IF PARTICIPANT CONSENTED; IF THEY DID NOT, BOTH THE INTERVIEWER AND NOTETAKER WILL TAKE NOTES TO CAPTURE AS MUCH DATA AS POSSIBLE] I’m going to start the recording now.

Background

[INFORMATION ON NAME, CURRENT ROLE, AND NAME OF THE NOYCE PI TO THE PROJECT FOR WHICH RESPONDENT IS AN EVALUATOR FOR WILL BE CAPTURED FROM THE DEMOGRAPHIC QUESTIONNAIRE AND NSF DATA – SHARE THE RESPONDENTS’ DEMOGRAPHIC QUESTIONNAIRE ANSWERS AND VERIFY THEY ARE CORRECT.

IF ANY INFORMATION IS INCORRECT, PLEASE NOTE THE RESPONDENT’S CORRECTED INFORMATION AND GIVE THE UPDATED INFORMATION TO THE PROJECT LEAD OR CO-LEAD]

FACILITATOR SCRIPT: First, we will talk a little bit about your background as an evaluator for a Noyce project.

1. How long have you been an evaluator for a Noyce project? Please consider all institutions with which you've worked. [BACKGROUND]

Evaluating Noyce Programs

2. How do you approach measurement and evaluation for the Noyce project you currently support? [1A]

   1. Probe: For those of you supporting Tracks 1-3, what do you consider when thinking about the Noyce Program’s effectiveness at retaining Noyce-eligible STEM majors and/or professionals in teaching careers in high-need schools?

   2. Probe: For those of you supporting Tracks 1-3, how do you measure Noyce recipients’ effectiveness as teachers/teacher leaders?

   3. Probe: How does your PI use the information from the evaluation to make adjustments, for example, to the project, if at all?

   4. Probe: For those of you supporting Tracks 1-3, what are your perceptions of the scholar/fellow data tracking and reporting requirements of the Noyce Award?

Perceptions of the Noyce Program

FACILITATOR SCRIPT: In this next set of questions, we would like to explore your perceptions about the Noyce Program overall and its impact as well as any gaps in its reach.

3. What is your overall perception of the Noyce Program? [2A]

   1. Probe: What is your perception of current Noyce scholars/fellows? [2B]

   2. Probe: What is your perception of former Noyce scholars/fellows? [2C]

4. What, if anything, could be done by the NSF and the Noyce Program overall to further support the evaluation of individual Noyce projects? [1A]

FACILITATOR SCRIPT: We’d now like to transition to talking about your experience with your team’s Noyce project application process.

5. With which entities do you collaborate as part of your team’s Noyce project? [IF NEEDED: NON-PROFIT PARTNERS THAT PROVIDED PROFESSIONAL DEVELOPMENT, INTERNSHIPS, RESEARCH EXPERIENCES; OTHER INSTITUTIONS; OTHER DEPARTMENTS; LOCAL SCHOOL DISTRICTS; ETC.] What role do they have in the evaluation? How did you develop these relationships? [1B, 2A]

1. Probe: Which entities are missing from your existing partnerships that would help support the evaluation of your team’s Noyce project? [1B, 2A]

6. What motivated you to be an external evaluator for your current Noyce project? What factors influenced your decision? In what ways did these factors influence your decision? For example, what factors existed within society, the larger field, at the institution level, and/or at the college, department, and faculty level? [1B, 2A]

7. How many of you have gone through the Noyce application process? For those of you who were involved in the application process, what are some specific challenges or complexities that you encountered during the application phase? For example, what challenges or complexities existed in terms of the amount of funding allowable for the evaluation portion of the Noyce project? [1B, 1C, 3, 3B, 3C, 3D]

FACILITATOR SCRIPT: As you know, one of the main intentions of the Noyce Program is to address the critical need for recruiting, preparing, and retaining highly effective elementary and secondary mathematics and science teachers and teacher leaders who persist as classroom teachers in high-need school districts.

To fulfill their teaching requirement for Noyce funding, scholars and fellows must be placed in a school in a high-need K-12 school district.

8. From what you have seen, are scholars and fellows being placed in what you’d consider to be a high-need school or school district? Why or why not?

1. Probe: What is your perception of the impact of the Noyce Program (i.e., after implementation)? For example, how does the Noyce Program address STEM teacher shortages and/or retaining STEM teachers in high-need schools? [2D]

2. Probe: What are the gaps in the Noyce Program's reach and impact to addressing the critical need for recruiting, preparing, and retaining highly effective STEM teachers and teacher leaders in high-need school districts.)? [2D]

FACILITATOR SCRIPT: For the next few questions, we will discuss your views on the merit review and funding process for the award.

9. How many of you have gone through the Noyce application merit review process? To what extent have you been involved in the merit review process? [4, 4B]

1. Probe: What was that process like? [4, 4B]

2. Probe: Overall, how satisfied are you with the merit review process, and what changes, if any, would you recommend? [4, 4B]

10. How many of you have gone through the Noyce declination process? What do you think of how the declination process is conducted? [4, 4B]

Wrap Up

11. As we wrap up our time together, would you like to share any lessons learned during your involvement with the Noyce Program or any growth opportunities that could help improve the Program? [5]

FACILITATOR SCRIPT: Thank you for your time and for sharing your experiences and perceptions. This will be extremely helpful for NSF as they identify areas in Noyce that are working well and opportunities for growth. As I mentioned at the beginning of our focus group, all information that you provided will remain confidential, and any identifying details will be removed before any information is shared with external parties. Finally, all of the recordings will only be made accessible to the WKC Evaluation Team.