Noyce Evaluation_SS A_draft2_to NSF_Clean

Noyce Evaluation_SS A_draft2_to NSF_Clean.docx

Evaluation of the National Science Foundation's Robert Noyce Teacher Scholarship Program

OMB:

Document [docx]
Download: docx | pdf




The Evaluation of the National Science Foundation’s (NSF) Robert Noyce Teacher Scholarship Program

Supporting Statement A: Justification

OMB No. 3145-New


DATE


Dr. Jennifer Ellis

Program Director

Division of Undergraduate Education

National Science Foundation

2415 Eisenhower Ave

Alexandria, Virginia 22314

Telephone: 703-292-8670

Email: [email protected]

Table of Contents



Attachments

1. Demographic Questionnaire

2. Data Collector Confidentiality Agreement Form

3a. Former Scholar/Fellow Focus Group Protocol

3b. Evaluators Focus Group Protocol

3c. Faculty Focus Group Protocol

3d. K-12 District and School Leaders Focus Group Protocol

3e. Principal Investigator Interview Protocol

3f. Potential Principal Investigator Interview Protocol

3g. Program Staff Coordinators Focus Group Protocol

3h. NSF Program Staff Focus Group Protocol

3i. Current Scholars and Fellows Focus Group Protocol

3j. AAAS Staff Focus Group Protocol

4a. Email from NSF to Principal Investigators Introducing the Evaluation

4b. Email from NSF to Current/Former Scholars/Fellows

4c. Reminder for Partial Demographic Questionnaire (All Respondent Types)

4d. Final Reminder for Partial Demographic Questionnaire (All Respondent Types)

5a. Questionnaire Invitation Email for Evaluators, Faculty, K-12 Leaders, and Program Coordinators

5b. Questionnaire Invitation Email for Principal Investigators

5c. Questionnaire Invitation Email for Potential Principal Investigators (Referred)

5d. Questionnaire Invitation for Potential Principal Investigators (Original Contact - Non-Noyce Institutions)

6a. Reminder - Questionnaire Invitation for Evaluators, Faculty, K-12 Leaders, and Program Coordinators

6b. Reminder - Questionnaire Invitation for Principal Investigators

6c. Reminder - Questionnaire Invitation for Potential Principal Investigators (Referred)

6d. Reminder - Questionnaire Invitation for Potential Principal Investigators (Original Contact - Non-Noyce Institutions)

7a. Final Reminder - Questionnaire Invitation for Evaluators, Faculty, K-12 Leaders, and Program Coordinators

7b. Final Reminder - Questionnaire Invitation for Principal Investigators

7c. Final Reminder - Questionnaire Invitation for Potential Principal Investigators (Referred)

7d. Final Reminder - Questionnaire Invitation for Potential Principal Investigators (Original Contact - Non-Noyce Institutions)

8a. Request for Principal Investigator to Forward Invitation Email (PI is Participating)

8b. Request for Principal Investigator to Forward Invitation Email (PI is Not Participating)

8c. Reminder - Request for Principal Investigator to Forward Invitation Email (PI is Not Participating)

8d. Final Reminder - Request for Principal Investigator to Forward Invitation Email (PI is Not Participating)

8e. Form to Collect Potential Participant Contact Information from PIs

9a. Focus Group Invitation for Evaluators, Faculty, K-12 Leaders, Program Coordinators, and Current/Former Scholars/Fellows

9b. Interview Invitation for Principal Investigators and Potential Principal Investigators

9c. Focus Group Invitation for AAAS Staff

9d. Focus Group Invitation for NSF Staff

10a. Reminder - Focus Group Invitation for Evaluators, Faculty, K-12 Leaders, Program Coordinator, and Current/Former Scholars/Fellows

10b. Reminder - Interview Invitation for Principal Investigators and Potential Principal Investigators

10c. Reminder - Focus Group Invitation for AAAS Staff

10d. Reminder - Focus Group Invitation for NSF Staff

11a. Final Reminder - Focus Group Invitation for Evaluators, Faculty, K-12 Leaders, Program Coordinator, and Current/Former Scholars/Fellows

11b. Final Reminder - Interview Invitation for Principal Investigators and Potential Principal Investigators

11c. Final Reminder - Focus Group Invitation for AAAS Staff

11d. Final Reminder - Focus Group Invitation for NSF Staff

12a. Scheduling Confirmation (All Focus Group and Interview Participants)

12b. Reminder and Meeting Info (All FG and Interview Participants)

12c. Thank You Email (All FG and Interview Participants)

12d. No-Show Follow-Up Email (All Participants)

13a. Newsletter/Social Media Post Recruitment Text

13b. Recruitment Flyer - Print Version

13c. Recruitment Flyer - Digital Version

14a. America COMPETES Act (P.L. 110-69)

14b. Higher Education Act of 1965 (20 U.S.C. 1021)

14c. National Science Foundation Authorization Act of 2002 (P.L. 107-368)

15a. Noyce Evaluation Questions and Subquestions

15b. Federal Register 60-Day Notice

15c. Federal Register 30-Day Notice

15d. Federal Register 30-Day Notice Public Comments

15e. Federal Register 30-Day Notice NSF’s Response to Public Comments

15f. NSF Statistical Contact feedback on Public Comments

15g. Estimates of Respondent Burden and Annualized Cost

16a. Questionnaire Invitation Phone Script – Evaluators, Faculty, K-12 Leaders, Program Coordinators

16b. Questionnaire Invitation Phone Script – Principal Investigators

16c. Questionnaire Invitation Phone Script – Potential Principal Investigators (Referred)

16d. Questionnaire Invitation Phone Script – Potential Principal Investigators (Original Contact – Non-Noyce Institutions)

17a. Focus Group Invitation Phone Script – Evaluators, Faculty, K-12 Leaders, and Program Coordinators

17b. Interview Invitation Phone Script – Principal Investigators and Potential Principal Investigators

17c. Focus Group Invitation Phone Script – AAAS Staff

17d. Focus Group Invitation Phone Script – NSF Staff

Part A: Justification

Introduction

The National Science Foundation (NSF) has been committed to broadening the participation of underrepresented groups and diverse institutions in science, technology, engineering, and math (STEM) through many means. One way that NSF supports this is through its Robert Noyce Teacher Scholarship Program (Noyce Program; https://ww.nsfnoyce.org), which was authorized in 2002 under the National Science Foundation Authorization Act of 2002 (P.L. 107-368; Attachment 14c.) and reauthorized in 2007 under the America COMPETES Act (P.L. 110-69; Attachment 14a.). The Program provides funding to higher education institutions via (a) scholarships, (b) stipends, and (c) other means of programmatic support to recruit and prepare STEM majors and other professionals to become K-12 STEM teachers.

The purpose of the proposed study is to understand what is working well and what are growth opportunities for its Noyce Program by administering up to 958 web-based demographic questionnaires, up to 70 virtual or in-person interviews, and up to 115 virtual or in-person focus groups, to further support its goal of increasing the diversity and “the number of K-12 teachers with strong STEM content knowledge who teach in high-need school districts.”1

A.1. CIRCUMSTANCES THAT MAKE DATA COLLECTION NECESSARY

This is a new information collection request. To support NSF’s ongoing efforts, components of the Noyce Program will be evaluated to (a) understand overall perceptions of the Program from both Noyce and non-Noyce recipients, and (b) identify what is working well and what are opportunities for growth.

The Robert Noyce Teacher Scholarship Program (Noyce Program) was authorized in 2002 under the National Science Foundation Authorization Act of 2002 (P.L. 107-368; Attachment 14c.) and reauthorized in 2007 under the America COMPETES Act (P.L. 110-69; Attachment 14a.).

This data collection is necessary to provide NSF with actionable information about the overall context of the Robert Noyce Teacher Scholarship Program, as well as to (a) understand overall perceptions of the Program from both individuals who have been part of a Noyce project and individuals who have not (i.e., have never applied for a Noyce award or have applied and not been funded), and (b) identify what is working well and what are opportunities for growth. This information will help support the overall goal of increasing the number and diversity of qualified teachers within the STEM education field. Further, the need is grounded in the importance and value of producing a diverse pool of STEM professionals who choose to pursue a career as a K-12 STEM teacher in a high-need school/school district.



Background

The overall goal of the Noyce Program is to “increase the number of K-12 teachers with strong STEM content knowledge who teach in high-need school districts” (The Robert Noyce Teacher Scholarship Program, https://www.nsfnoyce.org). Specifically, the Noyce Program’s goals include aiding in establishing and facilitating the awarding of scholarships, fellowships, funding, and programming that help to recruit, train, and retain K-12 STEM teachers for high-need school districts. Additional Noyce Program goals include supporting research on K-12 STEM teachers in high-need school districts to understand patterns of teacher retention and effectiveness in these settings. Program objectives for the Noyce Teacher Scholarship Program include increasing the (a) number and (b) diversity of students from groups underrepresented in STEM fields who pursue careers in K-12 STEM teaching, especially in high-need school districts.

There are four tracks that are funded:

  • Track 1: NSF Scholarships and Stipends (S&S)

  • Track 2: NSF Teaching Fellowships (TF)

  • Track 3: NSF Master Teaching Fellowships (MTF)

  • Track 4: Noyce Research

In addition to the four tracks, there is also a capacity-building grant for which PIs can apply. The intention of the capacity building grant is to support individuals in preparing for a future Noyce grant submission for Tracks 1-4.2

For Track 1, which provides scholarships and stipends as part of funding, those who receive these types of support are required to complete two years of teaching in a high-need school district for each of the years they receive the support. NSF defines “high-need school district” as articulated in Section 201 of the Higher Education Act of 1965 (20 U.S.C. 1021). That is, “a U.S. local educational agency (e.g., school district) that has at least one school that:

meets at least one of the following criteria:

  • not less than 20% of the children served by the agency are from low-income families;

  • serves at least 10,000 children from low-income families;

  • is eligible for funding under the Small, Rural School Achievement Program under 20 U.S.C. 7345(b); or

  • is eligible for funding under the Rural and Low-Income School Program under 20 U.S.C. 7351(b); and,

meets at least one of the following criteria:

  • has a high percentage of teachers not teaching in the academic subject areas or grade levels in which the teachers were trained to teach; or

  • has a high teacher turnover rate or a high percentage of teachers with emergency, provisional, or temporary certification or licensure.”3

For Track 2, which provides Fellowship support, there is a four-year teaching requirement for those individuals who receive support. During this four-year commitment, fellows receive supplements to their salary. For Track 3, which provides support to K-12 STEM teachers who hold either a bachelor’s or master’s degree in their field, there is a five-year teaching commitment. Track 4 requires a five-year teaching requirement in a high-need school district in return for salary supplements and professional development.4

The American Association for the Advancement of Science (AAAS; https://www.aaas-arise.org/) works with the Noyce Program to “identify and disseminate information about effective practices and strategies for attracting, selecting, and preparing new K-12 STEM teachers and retaining them in the STEM teacher workforce.” AAAS provides the following support to the Noyce Program:

  • Co-sponsoring and implementing the NSF Robert Noyce Teacher Scholarship Program annual conferences,

  • Highlighting the accomplishments of the NSF Noyce Teacher Scholarship Program and community through webinars, workshops, and social media,

  • Developing and maintaining an NSF Noyce Teacher Scholarship Program website, and

  • Organizing proposal preparation workshops.5

In 2023, the National Science Foundation (NSF) released a report indicating that while progress has been made to diversify the STEM field, women, Black, Hispanic, American Indian, and Alaska Native people and disabled individuals remain underrepresented when compared to the U.S. population overall.6 For example, the report indicated that in 2021, Black, Hispanic, American Indian, and Alaska Native people represented 31 percent of the U.S. population, but constituted only 24 percent of the STEM workforce. Diversifying the K-12 STEM teacher workforce can greatly impact the numbers of underrepresented students who matriculate to postsecondary institutions to pursue a degree in a STEM field. In turn, diversifying the K-12 STEM teacher workforce may increase the number of underrepresented students who pursue STEM careers.

Diversifying the STEM teacher workforce can have a significant impact on students’ education and future career pathways. Black students who have at least one Black teacher in elementary school are more likely to enroll in college than their Black peers who do not have a Black teacher.7 STEM teachers who reflect the diversity of the students they teach serve as role models (also referred to as “possibility models”), encouraging underrepresented students to pursue STEM fields themselves.

Campus-level STEM programs in colleges have been shown to provide educational support and interpersonal connections that assist underrepresented STEM students in persisting and completing STEM programs.8 A 2023 systematic review9 of diversity-focused STEM programming in the United States highlighted that successful interventions to increase the number of underrepresented students in STEM programs have several factors in common, including providing financial aid, facilitating opportunities to socialize with other STEM majors and receive support from STEM teachers, as well as offering high school-to-college bridge programming. To prepare future STEM teachers, it is necessary for these social and financial facilitators to continue to address barriers that may discourage college students from choosing to pursue a career teaching STEM teaching10 (e.g., rising tuition costs, certifications).

Since STEM teachers can have such a large impact on K-12 students, continued sponsored funding and programming for increasing STEM program persistence and completion is needed. Moreover, it is imperative to support students who wish to pursue STEM teaching careers to ensure the ongoing education and motivation of future STEM professionals in the United States.

This study represents continuing efforts by NSF to understand and improve ways to enhance and support quality STEM education in high-need school districts through its Robert Noyce Teacher Scholarship Program.

The Study

This study will utilize data collected from (a) a demographic questionnaire, and a series of semistructured (b) interviews, and (c) focus groups to assess perceive impact and effectiveness of the Noyce Program. The demographic questionnaire will largely garner information about individual-level characteristics to better inform the composition of the focus groups, utilize the most appropriate data collection instrument, and facilitate potential disaggregations of the qualitative data. The semistructured approach to the interviews and focus groups relies on a written list of questions and topics that are to be covered in a particular order, but also allow for the facilitators and respondents to explore new, relevant ideas and/or topics that may arise during data collection. This semistructured approach will enable NSF to identify what is working well with the Noyce Program, determine growth opportunities, and tailor any additional updates to its Noyce Program accordingly.

A.2. PURPOSE AND USE OF THE INFORMATION

a. Study Objectives

The purpose of this study is for program evaluation. Specifically, the purpose is to administer a demographic questionnaire and a series of semistructured interviews and focus groups to collect information on three overarching aspects related to the Robert Noyce Teacher Scholarship Program: (1) understanding the Noyce Program’s perceived impact and contributions to the larger STEM field, (2) uncover commonalities and nuances across the four tracks and potential impacts of individual and institution-level characteristics on experiences, and (3) provide an understanding of how applicants perceive the merit review process.

The findings from this study will be used to understand what is working well and what are growth opportunities for NSF to consider as possible refinements to the Noyce Program to continue addressing the need for high-quality teachers in high-need school districts. The data collection instruments are presented in Attachment 1. and Attachments 3a. through 3j.

b. From whom will the information be collected?

Six (6) out of the total of eleven (11) of data collection instruments were pretested: (1) the demographic questionnaire, (2) the Principal Investigator (PI) interview data collection instrument, (3) the Potential Principal Investigator (Potential PI) interview data collection instrument, (4) the current Noyce Scholars/Fellows focus group data collection instrument, (5) the External Evaluators focus group data collection instrument, and (6) the Program Support Staff/Program Coordinator focus group data collection instrument. The NSF Program Staff focus group data collection instrument, AAAS Staff focus group data collection instrument, former Noyce Scholars/Fellows focus group data collection instrument, Faculty focus group data collection instrument, and K-12 School Leaders focus group data collection instrument were not pretested, due to the challenges of recruiting respondents in these populations, which included time of year and semester timing. These pretesting respondents were contacted by email. Pretesting respondents evaluated the instruments for understandability (e.g., confusing wording or layout), missing questions/concepts, and length of time to answer. For the data collection instruments that were respondent type-specific interview or focus group protocols, five (5) PIs, three (3) Potential PIs, four (4) current Scholars/Fellows, one (1) Program Support Staff/Coordinator, and one (1) External Evaluator pretested their respective data collection instruments. Additionally, one (1) co-PI (included in the PI respondent type by definition and who also pretested the PI interview protocol data collection instrument), one (1) Potential PI, and one (1) External Evaluator (who also pretested the External Evaluator focus group protocol data collection instrument) pretested the demographic questionnaire. The Evaluation Advisory Committee (EAC), whose members reflect the range of respondents within this project, also provided feedback on instruments, ensuring they aligned with goals and accounting for differences among respondent types that may influence data collection. The EAC was established to ensure that the evaluation can meet its measurement approaches and outcomes, including by providing insights into what being in various Noyce roles entails and any challenges.

Approximately 852 total respondents are anticipated to participate partially or fully in this study (excluding pretest participants). The respondents are Principal Investigators (PIs), potential PIs (i.e., representatives who are eligible for NSF’s Robert Noyce Teacher Scholarship Program funding but have not received the funding), program coordinators/program staff, faculty mentors, current and former scholars/fellows, high-need K-12 school and district leaders, external evaluators, American Association for the Advancement of Science (AAAS) staff, and current and former NSF staff. The demographic questionnaire respondents will include up to 72 PIs with Noyce awards between FY2014 to FY2024, up to 29 representatives who are eligible for NSF’s Robert Noyce Teacher Scholarship Program funding but have not received it, up to 86 program coordinators/program staff, up to 173 faculty, up to 115 current scholars/fellows, up to 92 former scholars/fellows, up to 115 high-need K-12 school and district leaders, and up to 58 external evaluators (approximately 740 total respondents). The interviews will include up to 50 PIs with NSF’s Robert Noyce Teacher Scholarship Program awards between FY2014 to FY2024 and up to 20 representatives who are eligible for funding but have not received it (potential PIs), all of whom will also have completed the questionnaire (approximately 70 total). The focus groups will include up to 60 program coordinators/program staff, up to 120 faculty, up to 80 current scholars/fellows, up to 64 former scholars/fellows, up to 80 high-need K-12 school and district leaders, and up to 40 external evaluators, all of whom will have also completed the questionnaire. Additionally, up to 3 AAAS staff and up to 22 current and former NSF staff are expected to participate in focus groups (approximately 469 total respondents). Table A1 offers the definitions for each of the respondent types included in this study.

Table A1. Respondent Types and Definitions

RESPONDENT TYPE

DEFINITION OF RESPONDENT TYPE

Noyce Principal Investigators

Those PIs (or Co-PIs) who have received a Noyce award from any of the four tracks and may or may not have received a capacity-building grant. This encompasses both current Noyce Program PIs/Co-PIs and past PIs/Co-PIs who received a Noyce award.

Noyce Program Support Staff/Coordinators

Noyce Program support staff/coordinators who conduct program activities, organizational tasks, and communications for the day-to-day operation of a Noyce Program.

Noyce Faculty

Those faculty who have been part of a Noyce award but who have not served in a PI or Co-PI role. These could be STEM or education faculty who serve as mentors or supervisors for Noyce scholars/fellows. On the grant, they may have the title of Co-I or Senior Personnel, for example.

Current Noyce Scholars/Fellows

Students or trainees in their respective institution’s Noyce-funded program. These could be students currently enrolled at their institution or recent graduates.

Former Noyce Scholars/Fellows

Individuals who participated in their respective institution’s Noyce-funded Program and have completed their required teaching service.

Potential PIs

Deans, college leads, or other faculty at institutions that are eligible to apply for a Noyce award but either (a) have never applied or (b) have applied but did not receive an award.

K-12 District or School Leaders

Principals, Assistant Principals, mentor teachers, or other K-12 administrators who are involved with the Noyce Program at their school or district.

Noyce External Evaluators

Individual Noyce award evaluators, who are required as a condition of the award and provide measurement, assessment, and evaluation for the Noyce award.

Current and Former NSF Staff

NSF program officers or other program staff who have worked on the Noyce Program either currently or in the past.

AAAS Staff

AAAS staff who interface with or are otherwise involved with the NSF Noyce Program.

c. How will the information be collected?

All demographic questionnaires will be administered using the web-based platform Qualtrics. All semistructured interviews and focus groups will be conducted either (1) virtually using an online conferencing platform or (2) during the regularly scheduled regional conferences and national summits by NSF’s Contractor’s trained researchers.

NSF is interested in gathering perceptions of the Noyce Program from individuals at Noyce-eligible institutions. Thus, given that the purpose of the information collection is for program improvement, the Contractor will use a purposive sampling design with components of stratified random sampling and oversampling with institutions being the sampling unit. Noyce-eligible institutions, for the purpose of this information collection, are accredited institutions of higher education offering teacher education or residency programs that allow STEM majors to obtain teaching certifications, as outlined by NSF. Noyce-eligible institutions that have received a Noyce award between FY2014 and FY2024 (“Noyce institutions”) will be carefully selected using quota sampling for award type (i.e., Track) as the primary selection stratum. Additionally, Noyce-eligible institutions that have not received a Noyce award between FY2014-FY2024 (“non-Noyce institutions”) will be sampled. Both types of institutions will be sampled based on an institution type stratum (e.g., MSI, HSI, HBCU), oversampling for MSIs and HBCUs to ensure adequate representation and ability to achieve saturation.11 The institutions will then be randomly selected within the institution type stratum. One of the justifications for employing purposive sampling is to increase the coverage and depth of participants.12,13 Using the track and institution type strata can ensure shared variation across the respondents.14 From the initially identified sample, the institutional characteristics listed below will be reviewed to ensure diversity in representation:

  • Region (Desert and Pacific, Great Lakes, Great Plains, Interior Northeast, Mid-Atlantic, Mid-South, New York, Northeast, Southwest, West15),

  • Rurality,

  • Land grant status,

  • EPSCoR state status,

  • Percentage of undergraduates that complete an education degree,

  • Percentage of student body with a Pell Grant,

  • New to Noyce (for Noyce institutions only; an institution that has not had a new Noyce award in 5 years from the year of that award’s solicitation),

  • Public, Private, not-for-profit, and

  • Level (e.g., 2-year institution, 4-year institution).

These will not be proportional to the total number of Noyce institutions, but rather the Contractor will ensure that the institutions selected will include a range of characteristics to ensure the widest range of perspectives are captured.

These collections, to be conducted by the evaluation Contractor, include:

Demographic Questionnaire. A one-time, web-based survey of selected individuals who have been involved in various aspects of NSF’s Robert Noyce Teacher Scholarship Program from FY2014 to FY2024 (i.e., PIs, program coordinators, faculty, scholars, current and former scholars/fellows, K-12 school or district leaders, external evaluators) as well as potential PIs. An initial invitation email will be sent to selected PIs (Attachment 5b.) and potential PIs (Attachment 5c. and 5d.) approximately 2 weeks after OMB approval. This invitation email includes information about the study, requests their voluntary participation, and includes a link to the demographic questionnaire (Attachment 1.). Those PIs and potential PIs who do not respond to this first invitation email may receive a follow-up invitation email beginning 3 weeks after OMB approval as well as a final reminder email (Attachments 6b. and 7b. for PIs and Attachments 6c., 6d., 7c., and 7d. for Potential PIs). Those individuals who do not respond to any of the email invitations will be called one time 1 week after the final reminder is sent (for those with available telephone numbers), to invite them to participate (Attachments 16b. through 16d.).

To recruit the evaluator, faculty, K-12 school or district leader, current and former scholar/fellow, and program support staff/program coordinator respondent types, PIs will be asked to forward a recruitment email that includes a link to the demographic questionnaire. If PIs are not able to forward the recruitment email, they can provide referral information for the program support staff/ program coordinators, faculty, K-12 district or school leaders, and external evaluators in the last five minutes of the PI interview (Attachment 8e.). Subsequently, this referral information will be used to send initial recruitment emails (Attachment 5a.). Identical to the PIs and potential PIs, those who do not respond to this initial invitation email may receive a follow-up invitation email one week after the initial email invitation (Attachment 6a.), as well as a final reminder email if necessary (Attachment 7a.). Those individuals who do not respond to any of the email invitations will be called one time 1 week after the final reminder is sent (for those with available telephone numbers), to invite them to participate (Attachment 16a.). Current and former scholars/fellows will not receive a phone call reminder.

The current and former scholars/fellows will receive an informational email from NSF about the evaluation that includes a link to the demographic questionnaire (Attachment 4b.). As detailed above, they will also receive an informational email from their respective PI (Attachment 8a. or 8b.) that provides information about the evaluation and requests and includes a link to the demographic questionnaire. Additionally, social media posts (Attachment 13a.) and recruitment flyers (Attachments 13b. and 13c.) will be circulated on NSF, AAAS, and WKC social media accounts and websites to encourage participation in the evaluation.

The demographic questionnaire aims to (a) ensure that the participants recruited are diverse and (b) support the qualitative data analytic approaches. Information about the respondents captured in the questionnaire will include demographic data (e.g., race/ethnicity, gender), length of time with the institution, and experience with the Noyce Program (as applicable). The questionnaire data will enable NSF to supplement the information gleaned from the interviews and focus groups, encouraging further understanding of what is working well and growth opportunities for the Noyce Program.

Interviews with Principal Investigators. Once the PIs have completed the demographic questionnaire (Attachment 1.), they will receive an email to schedule their interview (Attachment 9a.). After they have scheduled a time for their interview, they will receive confirmation that their interview was successfully scheduled (Attachment 12a.). Interviews will be conducted with a purposive sample of up to 50 PIs or Co-PIs who had NSF’s Robert Noyce Teacher Scholarship Program funding between FY2014 and FY2024. The interviews will be conducted either in person during the regional Noyce conferences (in 2024 and 2025), annual Noyce Summit (in 2025), or using Microsoft Teams at a time convenient for the respondents. Both in-person and virtual interviews will utilize similar processes, and in-person participants will have a similar burden estimate as virtual participants. In-person participation recruitment will occur prior to conferences, and sessions will be scheduled before participants have arrived to reduce the burden on participants. Prior to in-person interviews with PIs, they will be sent a demographic questionnaire and invitations to schedule an interview, which will occur in person at the designated Noyce event. For those who agree to participate, they will also be informed of the time and location of their session. The interviews will elicit information regarding how the Noyce Program can support the STEM teacher preparation field, the impact of the award, barriers and facilitators to applying for Noyce funding, and how changes to the solicitation over the past 10 years have impacted the perception of the Noyce award as well as perceptions of the merit review process. This information will be triangulated with the other data collected and the information from the document reviews to provide a holistic understanding of the context of the Noyce Program, what is working well, and what are growth opportunities.

Participants will receive a scheduling confirmation email (Attachment 12a.) soon after scheduling and a reminder email (Attachment 12b.) 2-3 working days before their scheduled interview to confirm the interview date and time. Each interview, whether via video and/or audio virtually or in person, will last approximately 1 hour. PIs will be asked for their permission to record the interview (either through the conferencing platform, Microsoft Teams, or via a digital audio recorder for telephone and in-person interviews, respectively). The recording will be used for transcription purposes only. The interview questions are structured but include probes, allowing the trained interviewer and participant to explore other relevant topics as they arise. A second researcher will be present during the interview, primarily involved in recording the collected data from interviews and focus groups through notetaking and recording the session. After the interview is complete, the recording will be transcribed and uploaded into Dedoose, which is a qualitative analysis software package.

Interviews with Potential PIs. Once they complete the demographic questionnaire (Attachment 1.), Potential PIs will be invited to participate in a 60-minute interview with the goal of up to 20 Potential PIs completing the interview. This sample includes individuals who have never applied for Noyce funding and those who have applied for but not received Noyce funding. The interviews will be conducted either in person during the regional Noyce conferences (in 2024 and 2025), annual Noyce Summit (in 2025), or using Microsoft Teams at a time convenient for the respondents, and will elicit information about how the Noyce Program can support the STEM teacher preparation field, the impact of the award, barriers and facilitators to applying for Noyce funding, and how changes to the solicitation over the past 10 years have impacted the perception of the Noyce award as applicable to the representatives’ roles. Both in-person and virtual interviews will utilize similar processes, and in-person participants will have a similar burden estimate as virtual participants. In-person recruitment will occur prior to conferences, and sessions will be scheduled before participants have arrived to reduce burden on participants and the Contractor. Prior to in-person interviews with potential PIs, they will be sent a demographic questionnaire and invitations to schedule an interview. For those who agree to participate, they will also be informed of the time and location of their session.

Like the PIs, after they have scheduled a time for their interview, potential PIs will receive confirmation that their interview was successfully scheduled (Attachment 12a.). Likewise, potential PI participants will receive a reminder email (Attachment 12b.) 2-3 working days before their scheduled interview to confirm the interview date and time. Each interview, whether via video and/or audio virtually or in person, will last approximately 1 hour. Potential PIs will be asked for their permission to record the interview (either through the conferencing platform, Microsoft Teams, or via a digital audio recorder, for virtual and in person, respectively). The recording will be used for transcription purposes only. The interview questions are structured but include probes, allowing the trained interviewer and participant to explore other relevant topics as they arise. A second researcher will be present during the interview, primarily involved in recording the collected data from interviews and focus groups through notetaking and recording the session. After the interview is complete, the recording will be transcribed and uploaded into Dedoose, a qualitative analysis software package.

Focus groups. A series of up to 115 focus groups are also planned, including program coordinators/program support staff, faculty, current and former scholars/fellows, high-need K-12 school or district leaders, external evaluators, American Association for the Advancement of Science (AAAS) staff, and current and former NSF staff. This may include up to 469 total respondents. The focus groups will be conducted in person during the annual Noyce Summit (in 2025) or Noyce regional summits (2024-2025), or using Microsoft Teams at a time convenient for the respondents, and will elicit information about how the Noyce Program can support the STEM teacher preparation field, the impact of the award, barriers and facilitators to applying for Noyce funding, and how changes to the solicitation over the past 10 years has impacted the perception of the Noyce award, as well as perceptions of the merit review process (as applicable to the respondents’ roles). Both in-person and virtual focus groups will utilize similar processes, and in-person participants will have a similar burden estimate as virtual participants. In-person recruitment will occur prior to conferences, and sessions will be scheduled before participants have arrived to reduce the burden on participants. Prior to in-person focus groups, participants will be sent a demographic questionnaire and invitations to schedule a focus group. For those who agree to participate, they will also be informed of the time and location of their session.

Like the PIs and potential PIs, the program coordinators, faculty, external evaluators, current and former scholars/fellows, high-need K-12 school or district leaders, AAAS staff, and current and former NSF staff will receive confirmation that their focus group was successfully scheduled (Attachment 12a.). Additionally, they will receive a reminder email (Attachment 12b.) 2-3 working days before their scheduled focus group to confirm the date and time.

All focus groups, whether via video and/or audio virtually or in person, will last approximately 1 hour. All individuals participating in the focus group will be asked for their permission to record the focus group interview (either through the conferencing platform Microsoft Teams or via a digital audio recorder, for telephone and in person, respectively). If one person within the focus group does not agree to be recorded, detailed notes will be taken instead. The recording will be used for transcription purposes only. The focus group questions for all protocols are structured but include probes, allowing the trained facilitators and participants to explore other relevant topics as they arise. A second researcher will be present during each focus group. After the focus group is complete, the recording will be transcribed and uploaded into Dedoose, a qualitative analysis software package.

d. How frequently will the information be collected?

Each respondent in this study except for NSF and AAAS staff will complete one web-based demographic questionnaire (Attachment 1.) and all will complete either (a) one individual interview (for the PIs and potential PIs; Attachments 3e. and 3f., respectively) or (b) one focus group (for the program coordinators, faculty, external evaluations, current scholars/fellows, former scholars/fellows, high-need K-12 school or district leaders, AAAS staff, NSF staff; Attachments 3g., 3c., 3i., 3a., 3d., 3j., and 3h., respectively). There is a potential for follow-up questions after the interviews and focus groups if additional clarification is needed and if the respondent agrees to potential follow-up. However, this is minimized because the Contractor’s facilitators are trained to seek clarification during the data collection.

e. Will the information be shared with any other organizations inside or outside the NSF or the government?

The aggregated and analyzed data may be published and available to the public. The final report may be available on the NSF website. Possible additional dissemination venues include Noyce events, other conferences, publication audiences, or other professional communities.

The information garnered from this evaluation will help support the overall goal of increasing the number and diversity of qualified teachers within the STEM education field. Further, the need is grounded in the importance and value of producing a diverse pool of STEM professionals who choose to pursue a career as a K-12 STEM teacher in a high-need school/school district.

A.3. USE OF INFORMATION TECHNOLOGY AND BURDEN REDUCTION

NSF is committed to complying with the E-Government Act of 2002 to promote the use of technology. NSF and its Contractor will use a combination of web-based, telephone, and in-person modalities for contacting and coordinating with respondents and for data collection.

A.4. EFFORTS TO IDENTIFY DUPLICATION AND USE OF SIMILAR INFORMATION

Every effort has been made to avoid duplication. NSF has reviewed other studies and to our knowledge, there is no similar information available or being collected for the current timeframe. While there is another evaluation of the Robert Noyce Teacher Scholarship Program being conducted, it is focused on outcomes of the Noyce Program awards and, therefore, is distinct from this evaluation.

A.5. IMPACTS ON SMALL BUSINESSES OR OTHER SMALL ENTITIES

This is a one-time voluntary information collection request. All information requested or required is held to the minimum required for the intended use. Because the respondents are affiliated with either academic institutions or K-12 schools/districts, the Contractor does not anticipate that any small businesses or entities will be involved.

A.6. CONSEQUENCES OF COLLECTING THE INFORMATION LESS FREQUENTLY

This study is the most direct way to identify what is working well and growth opportunities for its Noyce Program by gaining the perspectives of those directly and indirectly involved with Noyce, further supporting its goal of increasing the diversity and “the number of K-12 teachers with strong STEM content knowledge who teach in high-need school districts.”16 This collection will reduce the burden in the long term by helping inform the next iteration of the Noyce Program. NSF is asking for responses from a minimum number of individuals across a wide array of respondent types while enabling enough sample size to afford the ability to disaggregate the data by selected institution- and/or individual-level characteristics. All respondents will complete the demographic questionnaire (except NSF and AAAS staff) and will participate in either an interview or focus group only once. Because this is a one-time data collection, collecting the information less frequently would obstruct NSF’s ability to determine what areas of the Noyce Program are working well and what are growth opportunities, to ultimately better serve the Noyce-eligible institutions, the PIs and potential PIs, and ultimately K-12 students and their educators.

A.7. Special Circumstances Relating to the Guideline of 5 CFR 1320.5

There are no special circumstances. This information collection is conducted in accordance with the guidelines in the Code of Federal Regulations, 5 CFR 1320.5.

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency

A.8.a Federal Register Notice and Comments

Notice of this study was published in the Federal Register on May 31, 2024 (Volume 89, No. 106, pages 47181-47184). No comments were received on this 60-day Notice. The 30-Day Notice for the study was published on May 31, 2024 at 89 FR 47171 and no comments were received.

A.8.b Consultations Outside the Agency

To ensure that the data collection materials are comprehensive, relevant, and reflective, the Contractor created and convened the Evaluation Advisory Committee (EAC), which serves as a team of external experts that will help steer the project by providing feedback on materials and instruments. The EAC provided feedback on instruments across respondent types. Because the committee members have a range of experience regarding the Noyce Program, they were able to offer insight regarding test instruments, including language, inclusivity, and applicability to various Noyce roles (PI, scholar/fellow, external evaluator, etc.). After documents were sent to the committee, members left remarks and suggestions for changes, which were incorporated into further edits of evaluation documents.

In addition, for the data collection instruments that were respondent type-specific interview or focus group protocol data collection instruments, five (5) PIs, three (3) Potential PIs, four (4) scholars/fellows, one (1) program support staff/coordinator, and one (1) external evaluator pretested their respective data collection instruments. Additionally, one (1) co-PI (included in the PI respondent type by definition and who also pretested the PI interview protocol data collection instrument), one (1) Potential PI, and one (1) external evaluator (who also pretested the External Evaluator focus group protocol data collection instrument) pretested the demographic questionnaire. These pretesters were consulted about the burden estimate of the data collection instruments and other aspects related to the instruments (e.g., clarity of instructions, clarity of questions). Respondents in the pretest represented the organizations shown in Table A2 below.

Table A2. Contact Information for Organizations that Participated in the Pretest

ORGANIZATION

ORGANIZATION ADDRESS

ORGANIZATION PHONE NUMBER

City University of New York (CUNY) Queens College

6530 Kissena Boulevard

Flushing, New York 11367

(718) 997-5000

Albany State University

504 College Drive

Albany, Georgia 31705

(229) 430-4600

University of Central Florida

4000 Central Florida Boulevard

Orlando, Florida 32816

(407) 823-2000

Virginia Commonwealth University

901 Park Avenue

Richmond, Virginia 23284

(804) 828-2227

Fort Hays State University

700 College Drive

Hays, Kansas 67601

(785) 628-3478

Pennsylvania Western University

250 University Avenue

California, Pennsylvania 15419

(724) 938-4000

Hinds Community College

608 Hinds Boulevard

Raymond, Mississippi 39154

(601) 857-3232

University of Texas at Rio Grande Valley

1201 West University Drive

Edinburgh, Texas 78539

(956) 665-3439


A.9. Explanation of Any Payment or Gift to Respondents

The respondents in this study will not receive an incentive payment or gift.

A.10. Assurance of Confidentiality Provided to Respondents

Although the findings from this study will be in the public domain (via the NSF website), identifiable individual-level information will not be disclosed to anyone but the select Contractor’s staff conducting this study. All electronic versions of the interview and focus group transcripts and recordings will be kept on the Contractor's secure server. All data from the demographic questionnaire will be likewise kept on the secure server. Data from the demographic questionnaire, the interviews, and the focus groups will be presented in aggregate form (with no fewer than 3 respondents in each analysis) and therefore cannot be linked back to any individual respondent. Raw data delivered to NSF after the evaluation will be void of identifiable information.

Statements of confidentiality and anonymity, as well as requests for permission to record, will be included at the beginning of the demographic questionnaire (Attachment 1.) and will be read at the start of each interview (Attachments 3e. and 3f.) and focus group (Attachments 3a. through 3d. and Attachments 3g. through 3j.). To ensure that personal information remains private, the Contractor requires that all its staff adhere to its confidentiality policy and sign a confidentiality agreement form prior to engaging in data collection activities (Attachment 2.).

Further, the Contractor will assign a unique ID number to each respondent and provide data to NSF with this ID number. A separate file will associate the ID number with personal information, which will be kept private by the Contractor and will not be shared with NSF. Once the contract is complete and all deliverables are accepted, the Contractor will keep all files containing private information on a secure server for at least three years.

Anonymity

Any identifying information mentioned during the interviews or the focus groups will be omitted from the final transcripts. Thus, no names or other identifying information will appear in the final transcripts, nor will they be used in data analysis.

Recordings

Interview and focus group recordings, whether via Microsoft Teams or a digital recorder, will not be shared with anyone outside of the study team, unless otherwise required by law, and will only be used to ensure the accuracy of the transcription. After the transcriptions are complete and have been verified, all recordings will be permanently deleted.

A.11. Justification for Sensitive Questions

This study does not contain questions of a sensitive nature.

A.12. Estimates of Hour Burden Including Annualized Hourly Costs

NSF and its Contractor have engaged with 17 pretest respondents across all the data collection instruments. NSF plans to contact an additional up to 1,360 potential respondents to complete the work. Out of this 1,360, approximately 827 will complete the demographic questionnaire and 539 respondents (not including pretest respondents) will go on to participate in the study in its entirety.17 A total of 855 are considered non-respondents [(9 former NSF staff; 846 program coordinators, faculty, external evaluators, current scholars/fellows, former scholars/fellows, high-need K-12 school leaders) (i.e., not completing both the demographic questionnaire and interview/focus group)]. The Contractor estimates 827.04 total annual burden hours and 1,366 total annual responses, not including the pretest responses. Attachment 15g. and Table A3 show the estimates of the respondent burden for the proposed data collection. These estimates reflect consultations with program officials, the agency’s prior experience in collecting data, and the pretesting of data collection instruments and protocols. Table A3 also shows the estimated annualized cost to respondents. It has been calculated using the "May 2023 National Occupational Employment and Wage Estimates” obtained from the Bureau of Labor Statistics website (http://www.bls.gov) and average GS-15-Step 5 for the current POs (Pay & Leave: Salaries & Wages - OPM.gov).



Table A3. Estimates of Respondent Burden and Annualized Cost

Respondent Type 

Collection Title

Total No. of Respondents in Category

Burden Hours Per Respondent

Total Hours Burden

Average Hourly Rate Estimate

Estimated Annual Cost

Pretest

PIs

Pretest Demographic Questionnaire

1

0.33

0.33

$ 70.66

$ 23.00

Potential PIs

1

0.33

0.33

$ 58.66

$ 19.00

External Evaluators

1

0.33

0.33

$ 70.66

$ 23.00

PIs

Pretest Principal Investigator Interview Protocol

5

1

5

$ 70.66

$ 353.00

Potential PIs

Pretest Potential Principal Investigator Interview Protocol

3

1

3

$ 58.66

$ 176.00

Current Scholars/Fellows

Pretest Scholars and Fellows Focus Group Protocol

4

1

4

$ 47.84

$ 191.00

Program Coordinators

Pretest Program Coordinator Focus Group Protocol

1

1

1

$ 37.12

$ 37.00

External Evaluators

Pretest External Evaluators Focus Group Protocol

1

1

1

$ 70.66

$ 71.00

Subtotal


17




$ 893.00

Demographic Questionnaire

PIs

Demographic Questionnaire

78

0.33

25.74

$ 23.00

$ 1,794.00

Potential PIs

56

0.33

18.48

$ 23.00

$ 1,288.00

Program Coordinators

94

0.33

31.02

$ 23.00

$ 2,162.00

Faculty

188

0.33

62.04

$ 23.00

$ 4,324.00

External Evaluators

63

0.33

20.79

$ 23.00

$ 1,449.00

Current Scholars/ Fellows

125

0.33

41.25

$ 23.00

$ 2,875.00

Former Scholars/Fellows

100

0.33

33

$ 23.00

$ 2,300.00

High Need K-12 School Leaders

125

0.33

41.25

$ 23.00

$ 2,875.00

Subtotal


829




$ 19,067.00

Interviews

PIs

Principal Investigator Interview Protocol

50

1

50

$ 70.66

$ 3,533.00

Potential PIs

Potential Principal Investigator Interview Protocol

20

1

20

$ 58.66

$ 1,173.00

Focus Groups

Program Coordinators

Program Coordinator Focus Group Protocol

60

1

60

$ 37.12

$ 2,227.00

Faculty

Faculty Focus Group Protocol

120

1

120

$ 70.66

$ 8,479.00

External Evaluators

External Evaluators Focus Group Protocol

40

1

40

$ 70.66

$ 2,826.00

Current Scholars/ Fellows

Current Scholars/Fellows Focus Group Protocol

80

1

80

$ 47.84

$ 3,827.00

Former Scholars/Fellows

Former Scholars/Fellows Focus Group Protocol

64

1

64

$ 47.84

$ 3,062.00

High Need K-12 School Leaders

High Need K-12 School/District Leaders Focus Group Protocol

80

1

80

$ 77.10

$ 6,168.00

AAAS Staff

AAAS Staff Focus Group Protocol

3

1

3

$ 42.24

$ 127.00

Current NSF Staff

Current and Former NSF Staff Focus Group Protocol

6

1

6

$ 66.82

$ 401.00

Former NSF Staff 

Current and Former NSF Staff Focus Group Protocol

16

1

16

$ 70.66

$ 1,131.00

Subtotal


539




$ 32,974.00

 

Total

1,385

--

830

--

$ 52,919.00


A.13. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers

There are no other costs to respondents beyond those presented in Section A.12.

A.14. Annualized Cost to Federal Government

The overall annualized cost to the respondents is estimated to be $50,455.00. The hourly wage estimates for completing the interviews mentioned in the burden hours table are based on information from the Bureau of Labor Statistics website (http://www.bls.gov) and average GS-15-Step 5 for the current POs (Pay & Leave: Salaries & Wages - OPM.gov). Specifically, the "May 2023 National Occupational Employment and Wage Estimates” were used. For Principal Investigators, faculty, external evaluators, and former NSF staff, 25-1000 “Postsecondary Teachers” was used; for this, the average hourly rate was calculated for a 9-month contract because an hourly rate was unavailable. For program coordinators, 25-9031 “Instructional Coordinators” was used; for representatives who are eligible for NSF’s Robert Noyce Teacher Scholarship Program funding but have not received it, 11-9033 “Education Administrators, Postsecondary” was used; for current and former scholars/fellows, 25-2000 “Preschool, Elementary, Middle, Secondary, and Special Education Teachers” was used (for this, the average hourly rate was calculated for a 9-month contract because an hourly rate was unavailable); for K-12 School Leaders, 11-9032 “Education Administrators, Kindergarten through Secondary” was used (for this, the average hourly rate was calculated for a 9-month contract because an hourly rate was unavailable); and for AAAS staff, 19-0000 “Life, Physical and Social Science Occupations” was used.

A.15. Explanation for Program Changes or Adjustments

This is a new submission. There is no request for program changes or adjustments.

A.16. Plans for Tabulation and Publication and Project Time Schedule

Recruitment for data collection will begin 2 weeks following OMB approval and run for approximately 10 months. Focus group and interview recordings will be transcribed as data collection is completed so that data analysis can begin in 11 months after OMB approval. The draft report will be submitted by the Contractor to NSF in Year 3, Month 8 of the contract, and presentation materials will be submitted to NSF in Year 3, Month 11 of the contract.

There will be two products resulting from the analysis of data: a final report and a presentation. Table A4 offers the planned data collection, analysis, and reporting schedule.

Table A4. Data Collection Schedule

ACTIVITY

TIMELINE (DAYS)

NSF provides a list of Noyce awards for PI recruitment

Pre-OMB clearance

NSF emails study overview to PIs and current/former scholars/fellows

OMB clearance + 7 days

PI recruitment list developed from NSF list and integrated with IPEDS data

OMB clearance + 14 days

WKC sends recruitment email with demographic questionnaire to PIs

OMB clearance + 14 days

Potential PI recruitment list developed from NSF list & public information

OMB clearance + 14 days

WKC sends demographic questionnaire invitation to potential PIs

OMB clearance + 14 days

Demographic questionnaire reminders 1, 2, and 3 (phone) for PIs and potential PIs

Demographic questionnaire invitation + 7 days, +14 days, +21 days

Data linkages – IPEDS and ElSi data and demographic questionnaire results

Demographic questionnaire completion + up to 14 days (on a weekly rolling basis depending on when participant completes questionnaire)

Recruitment materials shared via NSF, AAAS, and WKC networks

From recruitment through data collection

PIs receive email request to forward recruitment email to eligible participants

After PI completion of demographic questionnaire

Demographic questionnaire responses collected for Noyce Faculty, Program Coordinators, Current Scholars/Fellows, Former Scholars/Fellows, K-12 District and School Leaders, and External Evaluators

Up to 7 months following NSF sending invitation emails to current scholars/fellows and former scholars/fellows and PIs

Interview/focus group invitation schedule request sent to PIs, potential PIs, NSF staff, and AAAS staff

Contact information from PI and Potential PI demographic questionnaire, the NSF and AAAS + 7 days

(up to 10 months following PI recruitment list development from NSF list)

Interview/focus group reminders 1, 2, and 3 (phone)

Contact information from PI and Potential PI demographic questionnaire, the NSF and AAAS + 7 days, +14 days, +21 days

(up to 10 months following PI recruitment list development from NSF list)

Conduct PI and Potential PI interviews, NSF and AAAS focus groups

Up to 10 months following PI recruitment list development from NSF list


Final referral request made for Noyce Faculty, Program Coordinators, K-12 District and School Leaders, and External Evaluators from PIs

Rolling after completion of PI interviews

Demographic questionnaire invitation sent from WKC to Noyce Faculty, Program Coordinators, K-12 District and School Leaders, and External Evaluators Receiving contact information from final PI referral request after interviews +7 days

Receiving contact information from final PI referral request after interviews + 7 days

Demographic questionnaire reminders 1, 2, and 3 (phone)

Demographic questionnaire + 7 days, + 14 days, + 21 days

Focus group invitation schedule request sent to Noyce Faculty, Current Scholars and Fellows, Former Scholars/Fellows, Program Coordinators, K-12 District and School Leaders, and External Evaluators

Creation of focus groups + 7 days

Focus group reminders 1, 2, and 3 (phone)

Creation of focus groups + 7 days, + 14 days, + 21 days

Conduct focus groups for Noyce Faculty, Program Coordinators, K-12 District and School Leaders, and External Evaluators

Up to 10 months following PI recruitment list development from NSF list

Demographic Questionnaire collection complete

Completed 7 months following PI recruitment list development from NSF list

Interview and focus group data collection complete

Completed 10 months following PI recruitment list development from NSF list


Once all data collection is complete, analysis of the quantitative data and coding and analysis of the qualitative data will begin.

Quantitative Data Analysis. The quantitative data collection will include a demographic questionnaire that will gather individual-level information. Descriptives (e.g., frequencies, percentages, means, standard deviations) will be calculated for those who participate in the focus groups and interviews using IBM SPSS®. This information will describe the individuals represented in the qualitative findings. The Contractor will also use the data from the demographic questionnaire as descriptors in the qualitative data analysis. For example, it can be used to explore and present themes by various characteristics or make qualitative comparisons of themes based on the descriptors.

Qualitative Data Analysis. The interview and focus group recordings will be transcribed on an ongoing basis. The transcription files will be imported into Dedoose qualitative software for data management and coding. A priori themes will be generated based on the articulated constructs. Multiple individuals will read through and code the interview and focus group transcripts to ensure comprehensiveness and accuracy. A codebook will be refined to further analyze the data in Dedoose. This will also allow for collaboration across coders. Emerging themes will also be identified by carefully observing the patterns among all participants and those with shared characteristics (e.g., early career faculty, HBCU professors) via select disaggregations. Cross-cutting themes, as well as those by selected characteristics, questions, and subquestions will be analyzed, provided there is enough representation among the various groupings to maintain confidentiality.

Synthesis. The qualitative results will be triangulated to further explicate the findings. Aspects of the Noyce implementation status with elements of the guiding framework will be identified and aligned to articulate the root causes of the findings.

A.17. Reason(s) Display of OMB Expiration Date is Inappropriate

NSF plans to display the expiration date of OMB approval on all forms, questionnaires, and other data collection instruments associated with this information collection.

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to the certification statement.






1 Robert Noyce Teacher Scholarship Program (National Science Foundation, https://www.nsfnoyce.org)

2 Robert Noyce Teacher Scholarship Program (National Science Foundation, https://www.nsfnoyce.org)

3 Robert Noyce Teacher Scholarship Program (National Science Foundation, https://www.nsfnoyce.org)

4 Robert Noyce Teacher Scholarship Program (National Science Foundation, https://www.nsfnoyce.org)

5 ARISE – Advancing Research & Innovation in STEM Education of Preservice Teachers in High-Needs School Districts (http://www.aaas-arise.org)

6 National Science Foundation (2023). NSF’s NCSES releases report on diversity trends in STEM workforce and education. https://new.nsf.gov/news/diversity-and-stem-2023

7 Gershenson, S., Hart, C. M. D., Hyman, J., Lindsay, C., & Papageorge, N. W. (2018). The long-run impacts of same-race teachers. National Bureau of Economic Research, NBER Working Paper No. 25254. https://doi.org/10.3386/w25254

8 Burt, B. A., Stone, B. D., Motshubi, R., & Baber, L. D. (2023). STEM validation among underrepresented students: Leveraging insights from a STEM diversity program to broaden participation. Journal of Diversity in Higher Education, 16(1), 53–65. https://doi.org/10.1037/dhe000030

9 Palid, O., Cashdollar, S., Deangelo, S., Chu, C., & Bates, M. (2023). Inclusion in practice: A systematic review of diversity-focused STEM programming in the United States. International Journal of STEM Education, 10(2). https://doi.org/10.1186/s40594-022-00387-3

10 Feder, T. (2022). The US is in dire need of STEM teachers. Physics Today, 75(3), 25-27. https://doi.org/10.1063/PT.3.4959

11 Ames, H., Glenton, C., & Lewin, S. (2019). Purposive sampling in a qualitative evidence synthesis: A worked example from a synthesis on parental perceptions of vaccination communication. BMC Medical Research Methodology, 19, 1-9. https://doi.org/10.1186/s12874-019-0665-4

12 Guest, G. (2015). Sampling and selecting participants in field research. In H. R. Bernard & C. C. Gravlee, (Eds.), Handbook of methods in cultural anthropology (2nd ed.), (pp. 215-250). Rowman & Littlefield.

13 Suen, L. J. W., Huang, H. M., & Lee, H. H. (2014). A comparison of convenience sampling and purposive sampling. Hu li za zhi, 61(3), 105. https://doi.org/105. 10.6224/JN.61.3.105

14 Campbell, S., Greenwood, M., Prior, S., Shearer, T., Walkem, K., Young, S., Bywaters, D, & Walker, K. (2020). Purposive sampling: Complex or simple? Research case examples. Journal of Research in Nursing, 25(8), 652-661. https://doi.org/10.1177/1744987120927206

15 Informed by NSF's I-Corps Hub (National Science Foundation, https://new.nsf.gov/funding/initiatives/i-corps/view-hubsn)

16 Robert Noyce Teacher Scholarship Program (National Science Foundation, https://www.nsfnoyce.org)

17 The Demographic Questionnaire will not be administered to AAAS staff, current NSF staff, or former NSF staff. To be considered a respondent, AAAS staff, current NSF staff, and former NSF staff will only need to complete a focus group.




File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
AuthorRachel Messer
File Modified0000-00-00
File Created2024-11-14
© 2025 OMB.report | Privacy Policy