Overview of the Math and Science Partnership Program Data Monitoring System

National Science Foundation

Directorate for Education and Human Resources

Math and Science Partnership Program

Management Information System Support for the Math and Science Partnership Learning Network

Re-instatement with Change

Overview of the Math and Science Partnership Program Data Monitoring System

Division: Division of Undergraduate Education (DUE)

Contractor: Westat, Inc.

OMB Control # 3145-0199

Program Purpose

In 2002 Congress authorized the Math and Science Program under the National Science Foundation Authorization Act of 2002 (P.L. 107-368, Sec.10). The MSP program is a major research and development effort under the aegis of the America COMPETES Act of 2007 (Public Law 110-69). NSF’s MSP program also received a significantly increased appropriation in the American Recovery and Reinvestment Act of 2009 (Public Law 111-5). The goals for the program are to:

• Ensure that all K-12 students have access to, are prepared for, and are encouraged to participate and succeed in challenging curricula and advanced mathematics and science courses;

• Enhance the quality, quantity, and diversity of the K-12 mathematics and science teacher workforce; and

• Develop evidence-based outcomes that contribute to our understanding of how students effectively learn the knowledge, skills and ways of thinking inherent in mathematics, computer science, engineering, and/or the natural sciences.

To date, NSF has made over $965 million in commitments to partnership activities in a research and development portfolio that spans the nation. Individual projects funded by the MSP program aim to address the aforementioned issues by incorporating a depth and quality of creative strategic actions that extend beyond commonplace approaches. Although all MSP projects share a focus on the same set of fundamental issues, individual MSP projects differ in their scope and are categorized accordingly. Since NSF’s MSP program was initiated in FY 2002, awards have been made to the following types of projects:

• Comprehensive Partnerships implement change in mathematics and/or science educational practices in both Institutions of Higher Education (IHEs) and in schools and school districts, resulting in improved student achievement across the K-12 continuum.

• Targeted Partnerships focus on improved K-12 student achievement in a narrower grade range or disciplinary focus within mathematics or science.

• Institute Partnerships, also referred to as Teacher Institutes for the 21st Century, focus on the development of mathematics and science teachers as school- and district-based intellectual leaders and master teachers.

• Research, Evaluation, Technical Assistance (RETA) awards build and enhance large-scale research and evaluation capacity for all MSP awardees and provide them with tools and assistance in the implementation and evaluation of their work.

• MSP-Start Partnerships are for awardees new to the MSP program, especially from minority-serving institutions, community colleges and primarily undergraduate institutions, to support the necessary data analysis, project design, evaluation and team building activities needed to develop a full MSP Targeted or Institute Partnership.

• Phase II Partnerships for prior MSP Partnership awardees focus on specific innovation areas of their work where evidence of significant positive impact is clearly documented and where an investment of additional resources and time would produce more robust findings and results.

Major Evaluation Questions

This is a monitoring system only. The system was a primary data source for the previous MSP program evaluation (COSMOS, 2004 – 2012) and will continue to be the basis for any future program evaluation. In addition, the MSP MIS is the primary source of information for formation requests from program stakeholders (e.g., NSF, Congress, GAO, OMB, OSTP).

Research Design/Methods

This is a monitoring system only.

Ongoing Annual Program Monitoring

The program management information system was launched in 2004 and has been collecting project information for eight years. The MSP Management Information System is designed to collect both quantitative and qualitative data on an annual basis and will allow for analysis of standardized project descriptions and support comparisons across projects over time. The data collection is designed to facilitate the triangulation of information across the different groups of respondents and the aggregation of data within respondent groups. The MSP MIS is a monitoring system only and is designed to collect quantitative and qualitative data on an annual basis for comparisons both within and mong projects over time. Information from the MSP MIS is used for program management purposes and reporting purposes and will be used in response to the FY 2012 – FY 2013 Agency Priority Goal. As stated above, the MSP MIS will continue to be a primary data source for any future program level evaluations and for responding to information request from NSF, Congress, GAO, OMB, OSTP, and other program stakeholders. In addition, the MIS shall be the primary data source for a follow-up study that continues the trend analysis of student achievements, which was started during the previous program evaluation. This trend analysis will include an additional three years of data.

The primary method of gathering this information is by the use of a Web-based data collection system that incorporates the eight surveys including:

• Comprehensive and Targeted MSP Projects¹

• Annual Survey for Comprehensive and Targeted Partnership Projects. This survey collects information on each of the project's partner organizations (e.g., IHEs, K-12 school districts, project evaluators), the grades and subject areas the project will address, and project activities by key feature. It can be completed by the principal investigator (PI) or someone designated by the PI. Included in the survey is an Administrative Module that must be completed in order that K-12 district partners, IHE partners, and IHE participants can have access to the system to complete the survey modules for which they are responsible.

• Annual IHE Participant Survey for Comprehensive and Targeted MSPs. This survey module collects information about the characteristics and contributions of IHE faculty members and administrators who are active participants in an MSP Comprehensive or Targeted project. Information collected includes demographic characteristics, current fields of research and instruction, and contributions to their MSP. The survey must be completed by each individual IHE faculty member and administrator who is directly supported by the MSP grant and/or directly participated in the development or implementation of MSP-related activities during the previous school year. This survey has been modified through the addition of 5 items to collect information for the FY 2012-Fy 2013 NSF Agency Priority Goal for Undergraduate Programs, with no change in scope and burden hours.

• Annual Institution of Higher Education Survey. This survey, completed by each MSP IHE partner participating in a Comprehensive or Targeted partnership, obtains information on the number of individuals who developed and/or delivered MSP activities and the number of individuals who were recipients of MSP activities.

• Annual K-12 District Survey. This survey, completed by participating K-12 school districts in a Comprehensive or Targeted partnership, collects standardized data about each district and the participating K-12 schools within the district.  At the district level, the survey collects information about professional development, K-12 personnel involvement with the development and/or delivery of MSP activities and the number of participating schools within the district.  At the school level, the survey collects information about all mathematics and science teachers in the schools with significant MSP involvement, participating mathematics and science teachers, school enrollment, student course enrollment in mathematics and science courses, student achievement on statewide, criterion-based, mathematics and science accountability assessments, and school performance on AYP.

• Institute Projects

• Annual Survey for Institute Partnership Projects. This survey collects information on each of the project's partner organizations (e.g., IHEs, K-12 school districts, project evaluators), the scope of the project (e.g., grades and subject areas the project addresses and criteria for selecting teachers) and project activities by key feature. It also collects information about the demographic characteristics of students and teachers in the schools of the K-12 Institute enrollees. It can be completed by the principal investigator (PI) or someone designated by the PI. Included in the survey is an Administrative Module that must be completed in order that IHE participants can have access to the system to complete the survey modules for which they are responsible.

• Annual IHE Participant Survey for Institute MSPs. This survey collects information about the characteristics and contributions of IHE faculty members and administrators who are active participants in an Institute MSP project. Information collected includes demographic characteristics, current fields of research and instruction, and contributions to their MSP. The survey must be completed by each individual IHE faculty member and administrator who is participating in an Institute MSP.

• Initial Survey for K-12 MSP Institute Participants. This paper-based survey collects information about the characteristics of K-12 teachers and administrators prior to their participation in an Institute MSP project. Information collected includes demographic characteristics, school characteristics, instructional and administrative responsibilities, and educational preparation and certification. The survey must be completed by each K-12 teacher and administrator who is participating in an Institute MSP at the beginning of their participation in the program.

• Annual Survey for K-12 MSP Institute Participants. This survey collects information about the characteristics and professional development of K-12 Institute participants. Information collected includes participant’s current professional status, degrees and certifications earned, leadership responsibilities, Institute and professional development activities, and professional community building. This survey is completed annually by each individual K-12 Institute participant who has completed the Initial Survey for K-12 Institute Participant, are active in the MSP Institute program, and whose participation is directly funded by NSF MSP grant.

1 As continuations of prior partnership awards, Phase II projects also continue to complete the surveys for Comprehensive and Targeted Projects.