REQUEST FOR REINSTATEMENT OF
CLEARANCE FOR MONITORING FOR THE
NATIONAL SCIENCE FOUNDATION’S
(NSF’s)
MATH AND SCIENCE PARTNERSHIP (MSP) PROGRAM
This request for Office of Management and Budget (OMB) review asks for reinstatement of clearance of the online monitoring system for the Math and Science Partnership (MSP) program, which is funded by the Directorate of Education and Human Resources (EHR) at the National Science Foundation (NSF). The system is already in place, having been granted clearance for the first collection in 2004; this request asks for clearance for the next three (3) years of data collection. Since the last clearance, several changes have been made to the surveys to reduce respondent burden and address a new component of the MSP program. Specifically, several items have been removed (see A.15) from the Institute PI and K-12 targeted K-12 District surveys, the three new computer science courses have been added to the list of courses for which annual data are collected.
The NSF funds research and education in mathematics, science and engineering. It does this through grants, contracts, and cooperative agreements to more than 2,000 colleges, universities, and other research and/or education institutions in all parts of the United States. The Foundation accounts for about 24 percent of Federal support to academic institutions for basic research. EHR is the directorate within NSF that is responsible for the health and continued vitality of the Nation’s science, technology, engineering, and mathematics (STEM) education and for providing leadership in the effort to improve education in these areas.
Over the past 11 years, NSF has been using results from the online monitoring system to respond, in a timely fashion, to the Congressional mandate to provide ongoing program results on the MSP program. NSF also uses the data to monitor the annual activities and associated outcomes of individual projects. It also contributes substantially to the MSP project and program evaluations and provides important information for NSF’s K-12 reporting needs.
Timely clearance of this request is critical in order for NSF to continue meeting the Congressional mandate in P. L. 107-368, Sections 9 and 19 to evaluate the MSP program and provide Congress with ongoing results from this evaluation and for NSF to meet its Government Performance and Results Act (GPRA) accountability requirement.
A. Overview of the MSP Program
The MSP program is a major research and development effort under the aegis of the America COMPETES Act of 2010 (Public Law 111-358). To date, NSF has made over $1.2 billion in commitments to partnership activities in a research and development portfolio that spans the nation. The goals for the program are to:
Enhance schools’ capacity to provide challenging curricula for all students and encourage more students to succeed in advanced courses in mathematics and the sciences;
Increase the number, quality and diversity of mathematics and science teachers, especially in underserved areas;
Engage and support scientists, mathematicians, and engineers at local universities and local industries to work with K-12 educators and students;
Contribute to a greater understanding of how students effectively learn mathematics and science and how teacher preparation and professional development can be improved; and
Promote institutional and organizational change in education systems — from kindergarten through graduate school — to sustain partnerships’ promising practices and policies.
Individual projects funded by the MSP program aim to address the aforementioned issues by incorporating a depth and quality of creative strategic actions that extend beyond commonplace approaches. Although all MSP projects share a focus on the same set of fundamental issues, individual MSP projects differ in their scope and are categorized accordingly. Since NSF’s MSP program was initiated in FY 2002, awards have been made to the following types of projects:
Comprehensive Partnerships implement change in mathematics and/or science educational practices in both Institutions of Higher Education (IHEs) and in schools and school districts, resulting in improved student achievement across the K-12 continuum.
Targeted Partnerships focus on improved K-12 student achievement in a narrower grade range or disciplinary focus within mathematics or science.
Institute Partnerships, also referred to as Teacher Institutes for the 21st Century, focus on the development of mathematics and science teachers as school- and district-based intellectual leaders and master teachers.
Research, Evaluation, Technical Assistance (RETA) awards build and enhance large-scale research and evaluation capacity for all MSP awardees and provide them with tools and assistance in the implementation and evaluation of their work.
MSP-Start Partnerships are for awardees new to the MSP program, especially from minority-serving institutions, community colleges and primarily undergraduate institutions, to support the necessary data analysis, project design, evaluation and team building activities needed to develop a full MSP Targeted or Institute Partnership.
Phase II Partnerships for prior MSP Partnership awardees focus on specific innovation areas of their work where evidence of significant positive impact is clearly documented and where an investment of additional resources and time would produce more robust findings and results.
This clearance request covers a series of existing online surveys that are designed to obtain annual data from principal investigators (PIs) and other program participants for MSP Targeted, Institute, and Phase II Partnerships.1
B. Overview of the Study Design
The MSP Management Information System is designed to collect both quantitative and qualitative data on an annual basis and will allow for comparisons both within and among projects over time. The primary method of gathering this information is by the use of a Web-based data collection system that incorporates the eight surveys covered by this submission (and provided in Attachments A through H)—including:
Comprehensive, Targeted and Phase II MSP Projects
Annual Survey for Comprehensive and Targeted Partnership Projects (Attachment A). This survey collects information on each of the project’s partner organizations (e.g., IHEs, K-12 school districts), the grades and subject areas the project will address, and project activities by key feature. It can be completed by the principal investigator (PI) or someone designated by the PI. Included in the survey is an Administrative Module that must be completed in order that K-12 district partners, IHE partners, and IHE participants can have access to the system to complete the survey modules for which they are responsible.
Annual IHE Participant Survey for Comprehensive and Targeted MSPs (Attachment B).This survey module collects information about the characteristics and contributions of IHE faculty members and administrators who are active participants in an MSP Comprehensive or Targeted project. Information collected includes demographic characteristics, current fields of research and instruction, and contributions to their MSP. The survey must be completed by each individual IHE faculty member and administrator who is directly supported by the MSP grant and/or directly participated in the development or implementation of MSP-related activities during the previous school year.
Annual Institution of Higher Education Survey (Attachment C). This survey, completed by each MSP IHE partner participating in a Comprehensive or Targeted partnership, obtains information on the number of individuals who developed and/or delivered MSP activities, the number of individuals who were recipients of MSP activities, and information about MSP-supported materials and strategies for undergraduate students.
Annual K-12 District Survey (Attachment D). This survey, completed by participating K-12 school districts in a Comprehensive or Targeted partnership, collects standardized data about each district and the participating K-12 schools within the district. At the district level, the survey collects information about professional development, K-12 personnel involvement with the development and/or delivery of MSP activities and the number of participating schools within the district. At the school level, the survey collects information about all mathematics and science teachers in the schools with significant MSP involvement, participating mathematics and science teachers, school enrollment, student course enrollment in mathematics and science/computer science courses, and school performance on AYP. (Attachment D-1 contains a version of the survey with track changes to reflect the deletion of items and addition of new courses.)
Institute Projects
Annual Survey for Institute Partnership Projects (Attachment E). This survey collects information on each of the project’s partner organizations (e.g., IHEs, K-12 school districts), the scope of the project (e.g., grades and subject areas the project addresses and criteria for selecting teachers) and project activities by key feature. It can be completed by the principal investigator (PI) or someone designated by the PI. Included in the survey is an Administrative Module that must be completed in order that IHE participants can have access to the system to complete the survey modules for which they are responsible. (Attachment E-1 contains a version of the survey with track changes to reflect the deletion of items.)
Annual IHE Participant Survey for Institute MSPs (Attachment F). This survey collects information about the characteristics and contributions of IHE faculty members and administrators who are active participants in an Institute MSP project. Information collected includes demographic characteristics, current fields of research and instruction, and contributions to their MSP. The survey must be completed by each individual IHE faculty member and administrator who is participating in an Institute MSP.
Initial Survey for K-12 MSP Institute Participants (Attachment G). This survey collects information about the characteristics of K-12 teachers and administrators prior to their participation in an Institute MSP project. Information collected includes demographic characteristics, school characteristics, instructional and administrative responsibilities, and educational preparation and certification. The survey must be completed by each K-12 teacher and administrator who is participating in an Institute MSP at the beginning of their participation in the program.
Annual Survey for K-12 MSP Institute Participants (Attachment H). This survey collects information about the characteristics and professional development of K-12 Institute participants. Information collected includes participant’s current professional status, degrees and certifications earned, leadership responsibilities, Institute and professional development activities, and professional community building. This survey is completed annually by each individual K-12 Institute participant who has completed the Initial Survey for K-12 Institute Participant, are active in the MSP Institute program, and whose participation is directly funded by NSF MSP grant.
The MSP program is a major research and development effort that supports innovative partnerships to improve K-12 student achievement in mathematics and science. Cleared in 2013 for three years as OMB 3145-0199, a reinstatement of the MSP clearance that allows continued collection of data is requested for a total of eight surveys, listed above. While the content of the surveys has not changed, the following items have been removed to reduce respondent burden: (1) information about the characteristics of all students in K-12 schools participating in an Institute, and (2) information about student performance on statewide assessments in K-12 schools that meet the criteria for significant participation in Targeted projects. In addition, three new computer science courses have been added to the list of high school classes for which enrollment and completion data are being collected.
MSP projects are expected to both raise the achievement levels of all students and significantly reduce achievement gaps in the mathematics and science performance of diverse student populations. Successful projects will serve as models that can be widely replicated in educational practice to improve the mathematics and science achievement of all the nation’s students.
The MSP program is also directly aligned with two of NSF’s long-term investment categories that link directly to NSF programs and budget resources. They provide the framework for development of more specific and time-dependent performance goals, and for other assessments:
Transform the Frontiers: Investments that prepare and engage a diverse STEM workforce motivated to participate at the frontiers.
Innovate for Society: Investments that lead to results and resources that are useful to society. (http://www.nsf.gov/pubs/2014/nsf14043/nsf14043.pdf)
The MSP program represents an investment in the individual project participants and recipients of MSP activities, the institutions of higher education funded, and the collaborations fostered between K-12 schools and school districts and colleges and universities.
NSF goals and investment categories provide the framework for the development of NSF performance goals. Since the MSP program is a critical part of NSF’s efforts to meet these goals, the timely collection of data through the MSP Management Information System is essential for NSF’s documentation. In addition, the MSP Management Information System collects standardized information needed to evaluate the success of individual MSP projects and the MSP program as a whole. The 2015 MSP solicitation states that applicants for an MSP award must submit a proposal that includes:
“…a strategy for objective external review and feedback processes, including theoretical frameworks, any data collection plans, analysis plans, and reporting plans. Objective external feedback can be provided through an advisory board or through an independent external evaluator outside the proposing institution or in different organizational units than the PIs and Co-PIs. The external critical review or evaluation should be sufficiently independent and rigorous to influence the project’s activities, formatively, and improve the quality of its findings. Proposals should; (1) describe the expertise of the external reviewer(s); (2) explain how that expertise relates to the goals and objectives of the proposal; and (3) specify how the PI will report and use results of the project’s external, critical review process. Proposals must provide for a formative and summative evaluation that includes assessments of student/teacher learning outcomes and attitudinal changes, as appropriate” (NSF 15-537, p. 6,http://www.nsf.gov/pubs/2015/nsf15537/nsf15537.htm)
By collecting project-specific information that can be shared with the appropriate evaluators, the MSP Management Information System eliminates the redundancies associated with multiple evaluators developing their own data collection instruments to collect basic information needed for all of the individual project evaluations. At the same time, a single standardized data collection source provides the evaluation of the MSP program with data necessary to determine whether program objectives have been attained and to examine what project characteristics are most closely associated with project success.
MSP projects from the fourth, fifth, sixth, seventh, and eighth cohorts of the MSP program (as well as one Phase II project originally funded through the first three cohorts of the program) are currently being monitored by the online system. Continuing this data collection activity for these cohorts in future years, as well as monitoring the efforts of future cohorts, is necessary to decipher the extent to which programmatic outcomes are being achieved, as this monitoring system is the only method by which these data are being captured in a consistent manner across all awards.
The primary purpose for this data collection is program planning and management, also known as program monitoring, at the project and program levels. Monitoring the MSP program yields a better understanding of how the program is being implemented and its impact. NSF is using results in responding, in a timely fashion, to the Congressional mandate to provide ongoing program results on the MSP program. The monitoring data also provides important information for various NSF reports including the High Priority Performance Goals and various K-12 documents. Information collected will be used as a data source for the program evaluation. NSF has contracted with Westat, Inc., which developed the monitoring system for the MSP program. All information collected is and will continue to be used to provide analytical and policy support to EHR, assisting NSF to make decisions about future funding and other program initiatives to improve STEM education.
Westat provides NSF with annual reports displaying aggregated data for all MSP projects, as well as project-specific tables for each MSP project. Westat has also made electronic files available to individual MSP projects so they can review and extract their own data to facilitate their management and evaluation tasks. Project-specific data for all projects is available only to EHR staff, EHR contractors with responsibility for impact database management or program evaluators, and the NSF program managers and their staff.
The MSP data collection effort makes maximum use of computer technology to minimize the response burden and to maximize its ability to respond in a timely fashion to Congressionally mandated reporting requirements. Projects use a worldwide Web browser to submit the required data over the Internet using the specially developed MSP online data collection system software. EHR favors Web-based systems because they facilitate respondents’ data entry across computer platforms. One feature of the system is the thorough editing of data for completeness, validity, and consistency prior to final submittal. Editing is performed as data are entered. Questionable or incomplete entries are called to respondents’ attention before they are submitted to NSF. Features such as automatic tabulations, checkboxes, standard menus, and predefined charts and graphics facilitate the reporting process, provide useful and rapid feedback to the data providers, and reduce burden.
On the Annual IHE Participant Survey for Comprehensive and Targeted MSPs and the Annual IHE Participant Survey for Institute MSPs, certain items are only required of those respondents that participated for 40 or more hours in a given year (i.e., the system is designed to collect only a minimal amount of information from those IHE participants who participated less than 40 hours in their MSP project in any given year). The purpose is to obtain basic information on all IHE participants while minimizing response burden on those individuals who did not meet a specific threshold of participation. Similar procedures are in place in the Annual K-12 Survey to assure that more detailed information is only requested of those schools that have met a specific threshold of participation.
Furthermore, in each data collection period, individual items (e.g., contact information) show respondents’ data submitted in earlier years so that these data can be easily updated as opposed to re-entered. In addition, items that will never need to be revised or updated (e.g., type of organization that received the award) are not displayed in subsequent collections. Since most project participation is on a multi-year basis, updating the previous year’s data in subsequent collections is far easier and less burdensome than providing the data in the first year.
The online management information system is the only current means of collecting these data (no similar data exist elsewhere). All project data on program funding are drawn from the NSF administrative database now called the FastLane Project Reports system (OMB Control Number 3145-0058). Data collected via the monitoring system will be used, where possible, to pre-fill survey items in subsequent years to further minimize overall response burden.
No information is to be collected from small businesses.
Without the information collected via this monitoring system, NSF will be unable to document the effectiveness, impacts, or outcomes of the MSP program or meet its Congressionally mandated requirement under Public Law 107-368, subsection (c) (NSF Authorization Act of 2002) to have the MSP program evaluated and provide Congress with ongoing results from current evaluative activities or a future evaluation and will not meet its accountability requirements or assess the degree to which individual projects are meeting their goals.
The data collections will comply with 5 CFR 1320.6.
This data collection was published in Federal Register/Notices July 23, 2015 (Volume 80, Number 141 Page 43801-43802). A copy of the notice is attached in Attachment I.
No payments or gifts will be provided to respondents.
Respondents will be advised that no information on specific individuals will be maintained, in accordance with the Privacy Act of 1974. Data collected are available to NSF officials and staff, evaluation contractors, and the contractors hired to manage the data and data collection software. Data are processed according to Federal and State privacy statutes. Detailed procedures for making information available to various categories of users are specified in the Education and Training System of Records (63 Fed. Reg. 264, 272 January 5, 1998). That system limits access to personally identifiable information to authorized users. Data submitted will be used in accordance with criteria established by NSF for monitoring research and education grants and in response to Public Law 99-383 and 42 USC 1885c.The information requested may be disclosed to qualified researchers and contractors in order to coordinate programs and to a Federal agency, court or party in a court, or Federal administrative proceeding, if the government is a party.
The types of questions asked on these surveys are not considered sensitive. Furthermore, every effort has been made to protect the privacy of individuals involved in the MSP program. No individually identifiable information is collected by the surveys. The system has been designed so that neither Westat nor NSF will have information permitting them to identify MSP participants. Participants are listed in the online system using an identification number. The ID number is maintained by each MSP project, along with the individuals’ contact information. This information is used to track recipients of funding and training. Although the two Annual IHE Participant Surveys do ask for some demographic information, neither Westat nor NSF has the capability to link the information to any individual. Any public reporting of data is in aggregate form.
As mentioned above, the MSP Management Information System is designed to collect both quantitative and qualitative data on an annual basis that will allow for comparisons both within and among NSF-funded projects over time. All eight surveys are administered via an online system. Four of these surveys collect data from individuals (i.e., Annual IHE Participant Survey for Comprehensive and Targeted MSPs, Annual IHE Participant Survey for Institute MSPs, Initial Survey for K-12 MSP Institute Participants, and Annual Survey for K-12 MSP Institute Participants) and are designed to obtain basic information about individual participants and their participation in MSP activities. In keeping with the NSF’s MSP program monitoring goals, all eight instruments are designed to collect data that are easily accessible to respondents.
We estimate that the total number of annual respondents will be 1,936. The estimated annual response burden is 17,727 hours. The number of respondents per survey instrument and burden hour calculations is presented in Chart 1. Burden hours per response are estimated on the basis of discussions with NSF, PIs, and Westat’s experience in administrating the surveys over the previous eight years. Assumptions for the number of respondents are based on the number of respondents for the 2013–14 collection cycle (the most recent collection cycle for which information is currently available). Information about changes between the current estimates and the estimates in our last request for MSP survey clearance is contained in Section A.15.
Chart 1.—Annual Burden Hours for MSP Monitoring System, by Type of Respondent
|
Number of Respondents |
Burden Hours per Respondent |
Annual Person-Hour Total* |
Targeted and Phase II projects |
|||
Annual Survey for Comprehensive and Targeted Partnership Projects - Principal Investigator |
37 |
50.00 |
1,850 |
Annual Targeted and Comprehensive MSP IHE Participant Survey |
386 |
0.83 |
320 |
Annual IHE Survey for Comprehensive and Targeted MSPs -- IHE Partners |
84 |
8.00 |
672 |
Annual K-12 District Survey |
277 |
50.00 |
13,850 |
Institute Projects |
|||
Annual Survey for Institute Partnership Projects - Principal Investigator |
12 |
50.00 |
600 |
Annual IHE Participant Survey for Institute MSPs |
116 |
0.83 |
96 |
Initial Survey for K-12 MSP Institute Participants |
947 |
0.25 |
237 |
Annual Survey for MSP Institute K-12 Participants |
77 |
1.33 |
102 |
Total respondents |
1,936 |
|
17,727 |
* = Number of Respondents x Burden Hours per Response
There are a total of eight survey forms that we are requesting clearance for use in this program monitoring effort: 1) the Annual Survey for Comprehensive and Targeted Partnership Projects, 2) the Annual IHE Participant Survey for Comprehensive and Targeted MSPs, 3) the Annual Institution of Higher Education Survey 4) the Annual K-12 District Survey, 5) the Annual Survey for Institute Partnership Projects, 6) the Annual IHE Participant Survey for Institute MSPs, 7) the Initial Survey for K-12 MSP Institute Participants, and 8) the Annual Survey for MSP Institute K-12 Participants. All eight are Web-based surveys.
The hour burden estimates by type of form are presented in Chart 2:
Chart 2.—OMB Burden Calculation for MSP Monitoring System, by Survey Form
|
Number of Respondents |
Number of Responses per Respondent (over 3 years) |
Burden Hours per Response |
Total
Hour Burden |
Annual Average Hour Burden** |
Targeted and Phase II projects |
|||||
Annual Survey for Comprehensive and Targeted Partnership Projects - Principal Investigator |
37 |
3 |
50.00 |
5,550 |
1,850 |
Annual Targeted and Comprehensive MSP IHE Participant Survey |
386 |
3 |
0.83 |
961 |
320 |
Annual IHE Survey for Comprehensive and Targeted MSPs -- IHE Partners |
84 |
3 |
8.00 |
2,016 |
672 |
Annual K-12 District Survey |
277 |
3 |
50.00 |
41,550 |
13,850 |
Institute Projects |
|||||
Annual Survey for Institute Partnership Projects - Principal Investigator |
12 |
3 |
50.00 |
1,800 |
600 |
Annual IHE Participant Survey for Institute MSPs |
116 |
3 |
0.83 |
289 |
96 |
Initial Survey for K-12 MSP Institute Participants |
947 |
1 |
0.25 |
237 |
237 |
Annual Survey for MSP Institute K-12 Participants |
77 |
3 |
1.33 |
307 |
102 |
Total respondents |
1,936 |
|
|
52,710 |
17,727 |
* = Number of Respondents x Number of Responses per Respondent x Burden Hours per Response
** = Total Hour Burden (over 3 years) / 3
The overall annual costs to respondents for burden hours are estimated to be $761,021. The hourly wage rates were based on information found in the Department of Education’s National Center for Educational Statistics Integrated Postsecondary Education Data System (+Table 3, Employees in Postsecondary Institutions, Fall 2011 and Salaries of Full-Time Instructional Faculty, 2010-11, http://nces.ed.gov/pubs2012/2012276.pdf) and the U.S. Department of Labor, Bureau of Labor Statistics, National Compensation Survey (Table 3, National Compensation Survey: Occupational Wages in the United States: 2010, http://www.bls.gov/ncs/ncswage2010.pdf).
Calculations are shown in Chart 3.
Chart 3.—Annualized Cost to Respondents, by Type of Respondent
|
Annual Average Hour Burden |
Estimated Hourly Wage Rate |
Estimated Cost to Respondents* |
Targeted and Phase II projects |
|||
Annual Survey for Comprehensive and Targeted Partnership Projects - Principal Investigator |
1,850 |
$50.07 |
$92,630 |
Annual Targeted and Comprehensive MSP IHE Participant Survey |
320 |
$50.07 |
$16,022 |
Annual IHE Survey for Comprehensive and Targeted MSPs -- IHE Partners |
672 |
$50.07 |
$33,647 |
Annual K-12 District Survey |
13,850 |
$41.25 |
$571,313 |
Institute projects |
|||
Annual Survey for Institute Partnership Projects - Principal Investigator |
600 |
$50.07 |
$30,042 |
Annual IHE Participant Survey for Institute MSPs |
96 |
$50.07 |
$4,807 |
Initial Survey for K-12 MSP Institute Participants |
237 |
$37.05 |
$8,781 |
Annual Survey for MSP Institute K-12 Participants |
102 |
$37.05 |
$3,779 |
Total respondents |
17,727 |
|
$761,021 |
* Annual Average Hour Burden x Estimated Hourly Wage Rate
There is no overall annual cost burden to respondents or record keepers that results from the MSP program other than the time spent responding to the surveys that are attached as attachments to this request.
It is usual and customary for individuals involved in education and training activities in the United States to keep descriptive records. The information being requested is from records that are maintained as part of normal educational or training practice. In order to be funded by NSF, institutions must follow the instructions in the NSF Grant Proposal Guide (GPG) that is cleared under OMB 3145-0199. The GPG requires that all applicants submit requests for NSF funding and that all active NSF awardees do administrative reporting via FastLane, an Internet-based forms system. Thus, the PIs, school districts, IHE partners and IHE participants who are the primary respondents to the individual MSP data collections tasks make use of standard office equipment (e.g., computers), Internet connectivity that is already required as a startup cost and maintenance cost under OMB 3145-0199, and free software (e.g., Netscape or Microsoft Explorer) to respond.
The actual annual cost of the collection in 2014 was $479,065 (shown in Chart 4). However, since this cost reflects several one-time programming tasks, the actual amount for the three year period should be slightly lower. As such, the total estimated cost to the government of all data collection, analysis, and reporting activities for program monitoring over the 3-year period is approximately $1,123,093.
Chart 4.—Estimated Annual Cost to the Federal Government of Collection (based on 2014 expenditures)
Personnel |
$129,858 |
Travel |
$37 |
Computing |
$49,802 |
Overhead |
$140,228 |
G&A and Fee |
$159,140 |
Total Costs |
$479,065 |
Chart 5 summarizes changes in annual hour burden for the system of surveys between what was requested in 2008 and what is currently requested.
Although there is an increase in the overall burden (as well as the burden on five of the eight surveys), the hourly burden associated with three of the eight surveys decreased (the burden on the remaining five surveys remained the same). The decrease in burden reflects the removal of several school-level items on the PI and K-12 Partner surveys that were deemed by NSF program staff to no longer be necessary and/or the same information could now be obtained on districts’ websites.
The increase in overall burden shown in Chart 6 can be attributed to an increase in the number of respondents on six of the eight surveys. For example, the number of PIs completing the Annual Survey for Targeted Partnership Projects (Attachment A) increased by 13 as a result of an increase in the number of Targeted MSP projects since the 2012 request. As a result of there being more MSP projects, the number of individuals completing the other surveys for Targeted projects (Attachments B, C, and D) have also increased over the past three years.
Chart 5.—Hour Changes in Task Burdens
Attachment |
Collection title |
Annual Burden Requested in 2012 |
Currently Requested Annual Burden |
Change in Burden |
A |
Annual Survey for Comprehensive and Targeted Partnership Projects |
1,320 |
1,850 |
+530 |
B |
Annual IHE Participant Survey for Comprehensive and Targeted MSPs |
283 |
320 |
+37 |
C |
Annual Institution of Higher Education Survey |
520 |
672 |
+152 |
D |
Annual K-12 District Survey |
12,224 |
13,850 |
+1,626 |
E |
Annual Survey for Institute Partnership Projects |
900 |
600 |
-300 |
F |
Annual IHE Participant Survey for Institute MSPs |
185 |
96 |
-89 |
G |
Initial Survey for Institute K-12 Participants |
67 |
237 |
+170 |
H |
Annual Survey for Institute K-12 Participants |
746 |
102 |
-644 |
|
Total |
16,245 |
17,727 |
+1,482 |
Chart 6.—Changes in Number of Respondents
Attachment |
Collection title |
Number of Respondents in 2012 Request |
Respondents in Current Request |
Change in Number of Respondents |
A |
Annual Survey for Comprehensive and Targeted Partnership Projects |
24 |
37 |
+13 |
B |
Annual IHE Participant Survey for Comprehensive and Targeted MSPs |
341 |
386 |
+45 |
C |
Annual Institution of Higher Education Survey |
65 |
84 |
+19 |
D |
Annual K-12 District Survey |
191 |
277 |
+86 |
E |
Annual Survey for Institute Partnership Projects |
15 |
12 |
-3 |
F |
Annual IHE Participant Survey for Institute MSPs |
223 |
116 |
-107 |
G |
Initial Survey for K-12 MSP Institute Participants |
267 |
77 |
-190 |
H |
Annual Survey for MSP Institute K-12 Participants |
561 |
947 |
+386 |
|
Total |
1,687 |
1,936 |
+249 |
Chart 7 summarizes the timeline for the data collections and annual contractor reports for the first three data collection efforts.
Chart 7.—MSP Work plan for 2015–16 school year
Data Collection Tasks |
Date to start |
Date to complete |
Web data collection |
10/2016 |
3/2017 |
Follow-up Phone Calls to PIs |
11/2016 |
5/2017 |
Final Cleaning and Validation of Databases |
3/2017 |
7/2017 |
Tabulations for Report |
8/2017 |
9/2017 |
Draft Report to NSF |
|
10/2017 |
Project Reports to NSF |
|
10/2017 |
Files for Program Evaluation |
|
10/2017 |
Additional Reports |
TBD |
|
Like many agencies, NSF is reducing its reliance on formal (i.e., traditional) publication methods and publication formats. Westat is conducting this third-party monitoring of the MSP program on behalf of NSF and is forbidden contractually from publishing results unless NSF has made a specific exception. In short, all products of the collections are the property of NSF. After the products are delivered, NSF determines whether the quality of the products deserves publication verbatim by NSF, i.e., NSF is the exclusive publisher of the information being gathered. Often it is only after seeing the quality of the information delivered by the study that NSF decides the format (raw or analytical) and manner (in the NSF-numbered product Online Document System (ODS) or simply a page on the NSF Web site) in which to publish.
Before the conclusion of the study, both NSF and the funded MSP projects may use preliminary data to improve management and performance. For example, data generated by this effort are expected to be inputs to other internal and external NSF reports. At this time, NSF has no set timeline for publishing interim reports from analyses of monitoring data.
Not Applicable
Not Applicable
This data collection will be a census of the universe of Targeted, Institute and Phase II projects that NSF currently funds and anticipates funding through the MSP program. Forty-nine awards (37 Targeted and Phase II partnership awards and 12 Institute awards) will be surveyed. Responding on behalf of these awardees during the 2013–14 school year will be an estimated 1,936 respondents who come from the following categories: PIs for partnership awards, K-12 school districts, IHE partners, and IHE participants.
At the time of this submission, the universe of Comprehensive, Targeted, Phase II and Institute projects consists of 49 MSP awards – 37 Targeted and Phase II partnership awards, and 12 Institute awards. All of these projects are being included in the monitoring system.
Similarly, all 386 IHE participants, 84 IHE partners, and all 277 K-12 school districts of Targeted and Phase II projects are included, as are all 116 IHE participants in Institute Projects and 947 K-12 participants in Institute Projects. Since the full universe for each of the populations of interest is included, no statistical sampling will be used. Chart 8 summarizes the universe and sample information.
Chart 8.—Size of Universe and Sample
Survey |
Population |
Universe Size |
Sample |
Annual Survey for Comprehensive and Targeted Projects |
PIs for Comprehensive, Targeted, and Phase II awards |
37 |
37 |
Annual IHE Participant Survey for Comprehensive and Targeted MSPs |
IHE participants for Comprehensive, Targeted, and Phase II MSPs |
386 |
386 |
Annual Institution of Higher Education Survey |
IHE partners of Comprehensive, Targeted, and Phase II MSPs |
84 |
84 |
Annual K-12 District Survey |
School district partners of Comprehensive, Targeted, and Phase II MSPs |
277 |
277 |
Annual Survey for Institute Projects |
PIs for Institute awards |
12 |
12 |
Annual IHE Participant Survey for Institute MSPs |
IHE participants for Institute MSPs |
116 |
116 |
Initial Survey for K-12 MSP Institute Participants |
K-12 Participants of Institute MSPs |
77 |
77 |
Annual Survey for MSP Institute K-12 Participants |
K-12 Participants of Institute MSPs |
947 |
947 |
Total |
1,936 |
1,936 |
|
With the exception on the IHE and K-12 participant surveys, we expect the response rate will be 100 percent. Based on our experience in previous years, we anticipate that the response rates for the IHE and K-12 participant surveys will be approximately 80 percent.
This proposed monitoring system involves eight Web-based surveys, all of which have already been approved for data collection. These are the surveys for PIs of Comprehensive and Targeted partnership awards, IHE participants in Comprehensive and Targeted partnerships awards, IHE partners, and K-12 school districts. This submission requests clearance to conduct these surveys for the 2015–16 through the 2018–19 collection cycles.
NSF understands that in the absence of longitudinal data for both control and experimental groups, that it is not possible to conclusively attribute changes in student achievement (e.g., course completion, performance on accountability assessments) to the MSP program and has not and will not attribute observed changes to the program. The system is, therefore, intended to obtain descriptive information about the MSP projects and their participants in the project. Given that the MSP Management Information System is designed as a monitoring system, this limitation is to be expected. Information collected will be used as a data source for the program evaluation.
A second limitation is that the system requests that projects provide annual narrative information on the implementation and impact of individual activities. Because this represents self-reported assessments of project impact, additional information may be needed to assess whether these improvements actually occurred. Nonetheless, as part of the ongoing monitoring process, these self-reported findings can be used to help NSF staff and independent evaluators identify potentially promising practices that warrant further study. In addition, to the extent possible, these narratives will be examined during the validation process to assess whether additional information is needed to reinforce projects’ claims of success.
This study is a census of all MSP projects and all participants within projects. No sampling methodology will be employed for respondent selection.
Not Applicable
Not Applicable
Not Applicable
Not Applicable
Data collected for the online monitoring information system are considered part of NSF administrative requirements for awardees and therefore need to be completed in order for projects to maintain current funding levels. Therefore, we anticipate a response rate of 100 percent for each survey cycle for most of the surveys. The exceptions are the IHE Participant Survey (where we achieved a response rate of 83.4percent of Comprehensive and Targeted MSP participants and 84.5 percent of Institute MSP participants for the 2013–14 collection cycle) and the Annual Survey for MSP Institute K-12 Participants (where we achieved a response rate of 83.4 percent).
Each MSP project’s PI will be responsible for ensuring that individual data are obtained from partners and participants in project activities (i.e., K-12 school districts, IHE partners, and IHE participants of the Comprehensive and Targeted MSP projects and the IHE participants of the Institute MSP projects). Each PI will have access to an up-to-date list located on the Web site that will indicate who still needs to respond to the survey.
PIs will be provided with several types of assistance to ensure that they understand the importance of the data collection effort, their responsibilities for providing the data, and the technical aspects of data submission. For example, throughout the data collection cycle, Westat will provide respondents with online and telephone support to help them navigate the web site and address specific content issues. The “Home” screen of each survey provides names and contact information of contractor staff to contact with questions and comments. Text throughout the online surveys provides screen-specific instructions, definitions, and guidelines for survey completion. A “Help” screen, accessible from any point in the surveys, provides navigational instructions and answers frequently asked questions. As the due dates for data collection approach, Westat will monitor the response patterns of individual projects to identify respondents that need additional prompting and/or assistance. Issues regarding item non-response will be handled through the Web-based system, which requires that all mandatory items be completed as a condition of final submittal to NSF. In addition, follow-up letters or emails are used to remind each project of the data collection. The letters are found in Attachment J.
We believe that estimates of a 100 percent response rate for the Partnership Project Surveys, IHE Partner Survey, and Initial K-12 Participant Survey, and an 80 percent or higher response rate for the IHE Participant Surveys, Annual K-12 Participant Survey, and K-12 District Survey, are consistent with the results from the previous data collection cycles. These results are presented in Chart 9.
Chart 9.—Actual Response Rates for First Data Collection Cycle and Expected Response Rates for Future Cycles
Attachment |
Collection title |
Response Rate for 2013–14 collection cycle |
Expected Response Rate for 2015–16 collection cycle |
A |
Annual Survey for Comprehensive and Targeted Partnership Projects |
100 percent |
100 percent |
B |
Annual IHE Participant Survey for Comprehensive and Targeted MSPs |
83.4 percent |
80 percent or higher |
C |
Annual Institution of Higher Education Survey |
90.5 percent |
100 percent |
D |
Annual K-12 District Survey |
83.4 percent |
80 percent or higher |
E |
Annual Survey for Institute Partnership Projects |
91.7 percent |
100 percent |
F |
Annual IHE Participant Survey for Institute MSPs |
84.5 percent |
80 percent or higher |
G |
Initial Survey for K-12 MSP Institute Participants |
42.9 percent |
100 percent |
H |
Annual Survey for MSP Institute K-12 Participants |
66.6 percent |
80 percent or higher |
This data collection system has been in operation for the past 11 years and any problems that may have arisen with system procedures have been handled during this time, providing a thorough test of the system’s success. Prior to being opened to respondents, the web instruments used were extensively tested by Westat programmers and beta-testers to ensure that the programming was properly implemented; the system will be tested again by Westat programmers before each subsequent year of data collection begins.
Agency Unit
Kathleen Bergin, (703) 292-5171
Philis Hauser, NSF, 703-292-5104
Contractor
Joy Frechtling, Westat, 301-517-4006
Westat
will be responsible for data collection and analysis under the
direction of Gary Silverstein,
301-251-2244.
1MSP Comprehensive Partnerships are no longer funded by the program, MSP-Start projects complete a separate set of surveys, and RETA projects no longer complete an online survey.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Statement |
| Author | Gavin Fulmer |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |