Addendum to the Robert Noyce Teacher Scholarship Evaluation

National Science Foundation

Directorate for Education and Human Resources

Robert Noyce Teacher Scholarship Program

New Survey for Master Teaching Fellows

September 2011

ADDENDUM TO OMB PACKAGE CONTROL: NO. 3145-0217

Overview of the Noyce Program Evaluation

Division: Division of Undergraduate Education (DUE)

Contractor: Abt Associates, Inc.

Program Purpose:

In 2002 Congress authorized the Robert Noyce Teacher Scholarship Program under the National Science Foundation Authorization Act of 2002 (P.L. 107-368, Sec. 10). The program was reauthorized in the America COMPETES Act (P.L. 110-69, Sec. 7030) and in the Reauthorization of America COMPETES Act in 2010 (P.L. 111-358). Through this Act, the Director of the National Science Foundation (NSF) is authorized to “carry out a program to award grants to institutions of higher education (or consortia of such institutions) to provide scholarships, stipends, and programming designed to recruit and train mathematics and science teachers.” In the America COMPETES Act, Congress expanded the program to include summer internships for prospective pre-service students, as well as Teaching Fellowships for STEM career-changers preparing to become teachers, and Master Teaching Fellowships for current science and mathematics teachers preparing to become Master Teachers. The Noyce program received additional funding from the American Recovery and Reinvestment Act of 2009. Recipients of Noyce Scholarships, Stipends and Fellowships are required to teach in high-need school districts. Within each funded project, the project leadership team is expected to include both STEM discipline faculty and education faculty working in collaboration with school districts and Master K-12 Teachers.

Specifically, the long-term performance goals of the program are to:

(1) Encourage talented science, technology, engineering, and mathematics (STEM) majors and professionals to become science and mathematics K-12 teachers in high need districts and

(2) Retain talented mathematics and science teachers in these settings.

A prior third-party comprehensive evaluation was completed in Spring 2010 by Dr. Frances Lawrenz, University of Minnesota. Evaluation reports and other documents are available at the Noyce Program Evaluation Website

(http://www.cehd.umn.edu/EdPsych/NOYCE/default.html)

Major Overarching Evaluation Questions:

The program evaluation has five major evaluation questions each with a set of sub-questions. Each major evaluation question is associated with a research design and type of analysis. Each sub-question is aligned with data sources (e.g., primary data collection and/or extant data).

1. What are the goals of Noyce awards and what activities do their teacher preparation programs use to recruit, select, prepare, and support Noyce recipients?

2. How do stakeholders perceive the Noyce award and Noyce recipients?

3. What are the characteristics of the schools in which Noyce recipients teach?

4. What are the relationships between the types of supports, activities, and training that Noyce recipients receive, the types of Noyce recipients, and the recipients’ plans to go into and stay in teaching and leadership roles?

5. What is the impact of Noyce on teacher recruitment and retention and on student achievement?

Research Design/Methods

As described above, the evaluation addresses five questions or focuses. The evaluation plan is to collect information on the implementation of the program and describe the relationship among program characteristics, financial incentives, and teacher plans to enter or stay in teaching and teacher leadership roles. The evaluators also plan to examine the impact of the Noyce grant on an IHE’s production of STEM certified teachers who teach in high-need districts. In addition, the evaluators plan to conduct a substudy using extant data to examine the impact of the Noyce Program on student achievement in math and/or science for students who have been taught by Noyce teachers.

The evaluation of the Noyce program includes two types of research design with three types of analyses:

• A descriptive research design using descriptive analyses is used to answer Questions 1, 2, and 3.   The study uses surveys and interviews (1) to describe types of activities for recruitment, selection, preparation, and support of recipients; stakeholder perceptions; and schools in which Noyce recipients teach and (2) to examine the relationships among types of support, activities, types of recipients, retention in teaching, and involvement in leadership roles.

• A descriptive research design is also used to answer Question 4.  This portion of the evaluation uses relational analyses to answer the questions and uses statistical techniques (e.g., multiple regression) to statistically control for other factors (i.e., climate of schools, district hiring practices).

A quasi-experimental design is used to answer Question 5.  This portion of the program evaluation uses impact analyses to answer the question. To assess the impact of the program, the study uses a “difference of differences” quasi-experimental design that also could be described as a short interrupted time series with matched comparison groups designed to assess the impact of the program on teacher recruitment, retention, and student achievement. This component of the evaluation utilizes extant data, such as data collected by the Noyce program monitoring system, state longitudinal teacher certification and employment data from a sample of states, and student achievement data from a sample of districts. The addition of the collection and analysis of student achievement data is a modification to the original contract. To support this effort, a contract modification will be issued by NSF.

(See Overview Exhibit 1: Research Questions by Data Sources for the alignment of major research questions with sub-questions.)

Ongoing Annual Program Monitoring

This data collection activity is designed to track the extent to which Noyce awards meet the objectives of the program and to provide data for the evaluation. ICF Macro administers and manages the annual data collection. This information is used by the program to document the progress of the program and to respond to information requests from program stakeholders (e.g., NSF, OMB, Congress, COV). At the project level, PIs are required to collect and report a standard set of information regarding their Noyce projects on an annual basis. PIs report information on post-secondary institutions, school districts, scholarship recipients, stipend recipients, fellowship recipients, internships, and post-scholarship/stipend/fellowship follow-up. The Noyce Program Monitoring is a data resource for addressing Evaluation Questions 2, 4, and 5.

Expected Contributions of Noyce Program Evaluation:

Accountability: The Program Evaluation will allow NOYCE to specify the impact of the program on the recruitment of talented science, technology, engineering, and mathematics (STEM) majors and professionals to become science and mathematics K-12 teachers in high-need districts and the retention of talented mathematics and science teachers in these settings.

Program Improvement/Learning: The NOYCE Program Officers will utilize findings and information about the process, best practices, and impact of the program to improve the NOYCE program through revising solicitations and communicating with PIs.

Inform/Lead the Field: NOYCE Program Officers will utilize and disseminate the evaluation findings about successful systematic approaches and best practices for university STEM departments and STEM teacher education programs to NOYCE project and non-NOYCE universities and programs that pursue goals similar to those of NOYCE. This evaluation will produce findings and methods that are needed to further the study of the impact of financial incentives on efforts to recruit and retain STEM majors and professionals in teaching in high-need districts. Finally, the findings from the program evaluation could contribute to the knowledge base on STEM teacher recruitment and preparation programs and how best to address the national need for more mathematics and science teachers.

Additional Survey for Master Teaching Fellows:

On June 13, the data collection effort and evaluation described above were approved without change by OMB Control Number: 3145-0217. This included collecting primary data via surveys and interviews with Principal Investigators, STEM and Education Faculty, Noyce Recipients, and K-12 Principals.

As part of the evaluation, we would like to add a new survey for Master Teaching Fellows (MTFs), a group of recipients newly added to the program in 2009. The MTF survey will be similar to the other recipient surveys for recipients who are teaching; it differs in its focus on the leadership activities expected of these more experienced teachers and, since MTFs were not supported by the Noyce Program in preparing for certification or their early teaching years, there are no survey questions about their teacher preparation program or support during early teaching.

A separate survey for the MTFs was not included in the original August 2010 submission. In that submission, data collection was expected to begin in fall 2010, and data for the MTFs was not yet available at that time.

The other new group of recipients added to the program in 2009 was Teaching Fellows (TFs). This group receives Noyce program support in preparing for teacher certification and their early years of teaching. Its members will be asked the same survey questions asked of other recipients (e.g., scholarship recipients, stipend recipients).¹ These survey questions were submitted and approved June 13, 2011 (OMB Package Control: No.3145-0217).

Exhibit 1 includes the research questions by data sources for the overall study, as well as the type of design analyses to be performed in response to each research question. Under the column labeled “Primary Data Collection, Master Teaching Fellows”, evaluation questions are associated with the MTF survey and design type and analysis.

Exhibit 1: Research Questions by Data Sources - Overview of an NSF Program Evaluation – The Robert Noyce Teacher Scholarship Program
Research Question	Data Sources																	Research Design/Type of Analyses
	Primary Data Collection											Secondary (Extant) Data Collection
	PIs	STEM Faculty		Principals		Noyce Recipients1		Master Teaching Fellows		Interns		Demographic Data		Noyce Monitoring Data		Administrative data2
1. What are the goals of Noyce awards and what activities do their teacher preparation programs use to recruit, select, prepare, and support Noyce recipients?																		Descriptive Study/Descriptive Analyses
a. What are the goals and objectives of the Noyce awards?	
b. What strategies do Noyce awardees use to recruit and select Noyce candidates?		
c. What activities and supports do teacher preparation programs that have Noyce awards use to prepare Noyce recipients to teach in general, and to teach in high-need schools, in particular?	
d. What activities do teacher preparation programs that have Noyce awards use to support Noyce completers once they are teaching?		
e. What activities do teacher preparation programs that have Noyce awards use to introduce Noyce interns to teaching mathematics and science as a career option?	
f. What activities do Noyce awardees use to train and support the development of Master Teaching Fellows?	
2. How do stakeholders perceive the Noyce award and Noyce recipients, and what activities do Noyce recipient engage in?																		Descriptive Study/Descriptive Analyses
a. What effects do PIs and STEM faculty believe the Noyce award has had on recruitment of STEM teachers, in terms of quantity, quality, and diversity, the retention of those teachers? How do they perceive the burdens imposed by the Noyce award?		
b. What are school principals’ perceptions of benefits and burdens imposed by the Noyce award for their K-12 schools?				
c, How do school principals perceive the qualifications and teaching performance of Noyce recipients?				
d. What are faculty responsibilities for preparing K-12 mathematics and science teachers? What are faculty/departments perceptions of the effects of the Noyce award on their departments?			
e. How do Noyce recipients perceive the preparation they received from their teacher preparation program in preparing them for teaching? What supports do they use and what activities do they engage in during teacher preparation? When did Noyce recipients first become interested in teaching? What are the reasons Noyce recipients give for leaving the program or for teaching or not teaching in high-need districts?															
f. How do Noyce interns perceive the influence of their participation in the Noyce Program on their decision to enter teaching and on their interest in math and science? How else has the Noyce Program influenced them?															
g. What supports do Noyce teachers use, and what leadership activities do they engage in?									
3. What are the characteristics of the schools in which Noyce recipients teach?																		Descriptive Study/Descriptive Analyses
a. What are the demographic characteristics of schools/districts at which Noyce recipients teach?													
How do Noyce recipients perceive the climate of their schools?
How do the schools in which Noyce recipients teach work with Noyce IHEs?					
4. What are the relationships between the types of supports, activities, and training that Noyce recipients receive, the types of Noyce recipients, and the recipients’ plans to go into and stay in teaching and leadership roles?																		Descriptive Study/Relational Analyses (multiple regression/statistical control)
a. How are the types of supports/ activities/ training, financial incentives, school/district characteristics, or other personal experiences related to Noyce recipients’ plans to enter and/or remain in teaching and leadership roles?															
5. What is the impact of Noyce on teacher recruitment, retention, and on student achievement?																		Quasi-Experimental Study: Impact Analyses. Difference of Difference (pre-post with comparison groups design)
a. How does an IHE’s receipt of a Noyce grant affect its production of certified or licensed STEM teachers?																	
b. How does an IHE’s receipt of a Noyce grant affect its production of certified or licensed STEM teachers that take teaching jobs in high-need districts?																	
c. How does an IHE’s receipt of a Noyce grant affect the persistence in teaching in high-need districts among the STEM graduates of its teacher certification program?																	
d. Among students in high-need schools, what is the impact of being taught by a teacher who has received a Noyce grant on students’ achievement scores?																	
1 Noyce recipient surveys will be administered to Noyce scholars, stipend recipients, teaching fellows. Noyce Interns and Master Teaching Fellows will each receive a separate survey. 2 State longitudinal teacher certification and employment data from a sample of states and student achievement data from a sample of districts.

1 A sample of all recipient types, including TFs and MTFs, will also be interviewed as specified in the original clearance package.

10

Addendum to OMB Package Control: No. 3145-0217