Overview of program evaluation

National Science Foundation

Directorate for Education and Human Resources

Robert Noyce Teacher Scholarship Program

Draft Responses to OMB Questions about the Robert Noyce Teacher Scholarship Program Evaluation– March 2012

Overview of the Noyce Program Evaluation

Division: Division of Undergraduate Education (DUE)

Contractor: Abt Associates, Inc.

Program Purpose:

In 2002 Congress authorized the Robert Noyce Teacher Scholarship Program under the National Science Foundation Authorization Act of 2002 (P.L. 107-368, Sec. 10). The program was reauthorized in the America COMPETES Act (P.L. 110-69, Sec. 7030) and in the Reauthorization of America COMPETES Act in 2010 (P.L. 111-358). Through this Act, the Director of the National Science Foundation (NSF) is authorized to “carry out a program to award grants to institutions of higher education (or consortia of such institutions) to provide scholarships, stipends, and programming designed to recruit and train mathematics and science teachers.” In the America COMPETES Act, Congress expanded the program to include summer internships for prospective pre-service students, as well as Teaching Fellowships for STEM career-changers preparing to become teachers, and Master Teaching Fellowships for current science and mathematics teachers preparing to become Master Teachers. The Noyce program received additional funding from the American Recovery and Reinvestment Act of 2009. Recipients of Noyce Scholarships, Stipends and Fellowships are required to teach in high-need school districts. Within each funded project, the project leadership team is expected to include both STEM discipline faculty and education faculty working in collaboration with school districts and Master K-12 Teachers.

Specifically, the long-term performance goals of the program are to:

(1) Encourage talented science, technology, engineering, and mathematics (STEM) majors and professionals to become science and mathematics K-12 teachers in high need districts and

(2) Retain talented mathematics and science teachers in these settings.

A prior third-party comprehensive evaluation was completed in Spring 2010 by Dr. Frances Lawrenz, University of Minnesota. Evaluation reports and other documents are available at the Noyce Program Evaluation Website

(http://www.cehd.umn.edu/EdPsych/NOYCE/default.html)

Major Evaluation Questions:

The program evaluation has five major evaluation questions each with a set of sub-questions. Each major evaluation question is associated with a research design and type of analysis. Each sub-question is aligned with data sources (e.g., primary data collection and/or extant data).

1. What are the goals of Noyce awards and what activities do their teacher preparation programs use to recruit, select, prepare, and support Noyce recipients?

2. How do stakeholders perceive the Noyce award and Noyce recipients?

3. What are the characteristics of the schools in which Noyce recipients teach?

4. What are the relationships between the types of supports, activities, and training that Noyce recipients receive, the types of Noyce recipients, and the recipients’ plans to go into and stay in teaching and leadership roles?

5. What is the impact of Noyce on teacher recruitment and retention and on student achievement?

Research Design/Methods

As described above, the evaluation addresses five questions or focuses. The evaluators collect information on the implementation of the program and describe the relationship among program characteristics, financial incentives, and teacher plans to enter or stay in teaching and teacher leadership roles. The evaluators examine the impact of the Noyce grant on an IHE’s production of STEM certified teachers who teach in high-need districts. In addition, the evaluators plan to conduct a substudy using extant data to examine the impact of the Noyce Program on student achievement in math and/or science for students who have been taught by Noyce teachers. .

The evaluation of the Noyce program includes two types of research design with three types of analyses:

• A descriptive research design using descriptive analyses is used to answer Questions 1, 2, and 3.   The study uses surveys and interviews (1) to describe types of activities for recruitment, selection, preparation, and support of recipients; stakeholder perceptions; and schools in which Noyce recipients teach and (2) to examine the relationships among types of support, activities, types of recipients, retention in teaching, and involvement in leadership roles.

• A descriptive research design is also used to answer Question 4.  This portion of the evaluation uses relational analyses to answer the questions and uses statistical techniques (e.g., multiple regression) to statistically control for other factors (i.e., climate of schools, district hiring practices).

A quasi-experimental design is used to answer Question 5.  This portion of the program evaluation uses impact analyses to answer the question. To assess the impact of the program, the study uses a “difference of differences” quasi-experimental design that also could be described as a short interrupted time series with matched comparison groups designed to assess the impact of the program on teacher recruitment, retention, and student achievement. This component of the evaluation utilizes extant data, such as data collected by the Noyce program monitoring system, state longitudinal teacher certification and employment data from a sample of states, and student achievement data from a sample of districts. The addition of the collection and analysis of student achievement data is a modification to the original contract. To support this effort, a contract modification will be issued by NSF.

(See Overview Exhibit 1: Research Questions by Data Sources for the alignment of major research questions with sub-questions)

Ongoing Annual Program Monitoring

This data collection activity is designed to track the extent to which Noyce awards meet the objectives of the program and to provide extant data for the evaluation. Managed by ICF Macro, this information is used to administer and monitor the progress of the program through Noyce-supported projects in the various institutions. At the project level, PIs are required to collect and report a standard set of information regarding their Noyce projects on an annual basis. PIs report information on post-secondary institutions, school districts, scholarship recipients, stipend recipients, fellowship recipients, internships, and post-scholarship/stipend/fellowship follow-up. The Noyce Program Monitoring is a data resource for addressing Evaluation Questions 2, 4, and 5.

Expected Contributions of NOYCE Program Evaluation:

Accountability: The Program Evaluation will allow NOYCE to specify the impact of the program on the recruitment of talented science, technology, engineering, and mathematics (STEM) majors and professionals to become science and mathematics K-12 teachers in high-need districts and the retention of talented mathematics and science teachers in these settings.

Program Improvement/Learning: The NOYCE Program Officers will utilize findings and information about the process, best practices, and impact of the program to improve the NOYCE program through revising solicitations and in communications to PIs.

Inform/Lead the Field: NOYCE Program Officers will utilize and disseminate the evaluation findings about successful systematic approaches and best practices for university STEM departments and STEM teacher education programs to NOYCE project and non-NOYCE universities and programs that pursue goals similar to those of NOYCE. This evaluation will produce findings and methods that are needed to further the study of the impact of financial incentives on efforts to recruit and retain STEM majors and professionals in teaching in high-need districts. Finally, the findings from the program evaluation could contribute to the knowledge base on STEM teacher recruitment and preparation programs and how best to address the national need for more mathematics and science teachers.

Exhibit 1: Research Questions by Data Sources - Overview of an NSF Program Evaluation – The Robert Noyce Teacher Scholarship Program
Research Question	Data Sources									Research Design/Type of Analyses
	Primary Data Collection					Secondary (Extant) Data Collection
	PIs	STEM Faculty	Principals	Noyce Recipients1	Interns	Demographic Data	Noyce Monitoring Data	Administrative data2
1. What are the goals of Noyce awards and what activities do their teacher preparation programs use to recruit, select, prepare, and support Noyce recipients?										Descriptive Study/Descriptive Analyses
a. What are the goals and objectives of the Noyce awards?	
b. What strategies do Noyce awardees use to recruit and select Noyce candidates?		
c. What activities and supports do teacher preparation programs that have Noyce awards use to prepare Noyce recipients to teach in general, and to teach in high-need schools, in particular?				
d. What activities do teacher preparation programs that have Noyce awards use to support Noyce completers once they are teaching?				
e. What activities do teacher preparation programs that have Noyce awards use to introduce Noyce interns to teaching mathematics and science as a career option?					
2. How do stakeholders perceive the Noyce award and Noyce recipients?										Descriptive Study/Descriptive Analyses
a. What effects do PIs and STEM faculty believe the Noyce award has had on recruitment of STEM teachers, in terms of quantity, quality, and diversity, the retention of those teachers? How do they perceive the burdens imposed by the Noyce award?		
b. What are school principals’ perceptions of benefits and burdens imposed by the Noyce award for their K-12 schools?			
c, How do school principals perceive the qualifications and teaching performance of Noyce recipients?			
d. What are STEM departmental/faculty responsibilities for preparing K-12 mathematics and science teachers? What are STEM faculty/departments perceptions of the effects of the Noyce award on their departments?		
e. How do Noyce recipients perceive the preparation they received from their teacher preparation program in preparing them for teaching? When did Noyce recipients first become interested in teaching? What are the reasons Noyce recipients give for leaving the program or for teaching or not teaching in high-need districts?							
f. How do Noyce interns perceive the influence of their participation in the Noyce Program on their decision to enter teaching and on their interest in math and science? How else has the Noyce Program influenced them?							
3. What are the characteristics of the schools in which Noyce recipients teach?										Descriptive Study/Descriptive Analyses
a. What are the demographic characteristics of schools/districts at which Noyce recipients teach?						
How do Noyce recipients perceive the climate of their schools?				
How do the schools in which Noyce recipients teach work with Noyce IHEs?			
4. What are the relationships between the types of supports, activities, and training that Noyce recipients receive, the types of Noyce recipients, and the recipients’ plans to go into and stay in teaching and leadership roles?										Descriptive Study/Relational Analyses (multiple regression/statistical control)
a. How are the types of supports/ activities/ training, financial incentives, school/district characteristics, or other personal experiences related to Noyce recipients’ plans to enter and/or remain in teaching and leadership roles?							
5. What is the impact of Noyce on teacher recruitment, retention, and on student achievement?									Quasi-Experimental Study: Impact Analyses Difference of Difference (pre-post with comparison groups design)
a. How does an IHE’s receipt of a Noyce grant affect its production of certified or licensed STEM teachers?								
b. How does an IHE’s receipt of a Noyce grant affect its production of certified or licensed STEM teachers that take teaching jobs in high-need districts?								
c. How does an IHE’s receipt of a Noyce grant affect the persistence in teaching in high-need districts among the STEM graduates of its teacher certification program?								
d. Among students in high-need schools, what is the impact of being taught by a teacher who has received a Noyce grant on students’ achievement scores?								
1 Noyce recipient surveys will be administered to Noyce scholars, stipend recipients, and fellows. Noyce Interns will receive a separate survey. 2 State longitudinal teacher certification and employment data from a sample of states and student achievement data from a sample of districts.

Revised DRAFT response to OMB comments for NSF Evaluation of the Noyce Program 4