31450199 Supporting Statement Part B

Supporting Statement (3145-0199)

REQUEST FOR RENEWAL OF CLEARANCE FOR MONITORING FOR THE NATIONAL SCIENCE FOUNDATION'S (NSF's) MATH AND SCIENCE PARTNERSHIP (MSP) PROGRAM

Section B

Introduction

This data collection will be a census of the universe of projects that NSF currently funds and anticipates funding through the MSP program. Sixty-three awards (39 Comprehensive and Targeted partnership awards, 12 Institute awards, and 12 RETA awards) will be surveyed. Responding on behalf of these awardees during the 2008-09 school year will be an estimated 3,149 respondents who come from the following categories: PIs for partnership awards, K-12 school districts, IHE partners, IHE participants, and PIs for RETA awards.

B.1. Respondent Universe and Sampling Methods

At the time of this submission, the universe of NSF projects consists of the 63 MSP awards. These consist of 39 Comprehensive and Targeted partnership awards, 12 Institute awards, and 12 RETA awards. All of these projects are being included in the monitoring system.

Similarly, all 1,250 IHE participants, 134 IHE partners, and all 692 K-12 school districts of Comprehensive and Targeted projects are included, as are all 150 IHE participants in Institute Projects and 660 K-12 participants in Institute Projects. Since the full universe for each of the populations of interest is included, no statistical sampling will be used. Chart 9 summarizes the universe and sample information.

Chart 9. Size of Universe and Sample

Survey	Population	Universe Size	Sample Size
Annual Survey for Comprehensive and Targeted Projects	PIs for Comprehensive and Targeted awards	39	39
Annual IHE Participant Survey for Comprehensive and Targeted MSPs	IHE participants for Comprehensive and Targeted MSPs	1,250	1,250
Annual Institution of Higher Education Survey	IHE partners of Comprehensive and Targeted MSPs	134	134
Annual K-12 District Survey	School district partners of Comprehensive and Targeted MSPs	692	692
Annual Survey for Institute Projects	PIs for Institute awards	12	12
Annual IHE Participant Survey for Institute MSPs	IHE participants for Institute MSPs	150	150
Initial Survey for K-12 MSP Institute Participants	K-12 Participants of Institute MSPs	200	200
Annual Survey for MSP Institute K-12 Participants	K-12 Participants of Institute MSPs	660	660
Annual Survey of RETA PIs	PIs for RETA awards	12	12
Total		3,149	3,149

 

With the exception on the IHE and K-12 participant surveys, we expect the response rate will be 100 percent. Based on our experience in previous years, we anticipate that the response rates for the IHE and K-12 participants surveys will be approximately 80 percent.

B.2. Information Collection Procedures/Limitations of the Study

This proposed monitoring system involves eight Web-based surveys and one paper survey, all of which have already been approved for data collection. These are the surveys for PIs of Comprehensive and Targeted partnership awards, IHE participants in Comprehensive and Targeted partnerships awards, IHE partners, K-12 school districts, and PIs for RETA awards. In addition to requesting clearance to conduct these surveys in years 5 through 7 of the data collection, this submission includes a request for clearance of two additional items—one on the IHE Participant Survey and another on the K-12 District Survey.

NSF understands that in the absence of longitudinal data for both control and experimental groups, that it is not possible to conclusively attribute changes in student achievement (e.g., course completion, performance on accountability assessments) to the MSP program and has not and will not attribute observed changes to the program. The system is, therefore, only intended to obtain descriptive information about the MSP projects and their participants in the project. Given that the MSP Management Information System is designed as a monitoring system, this limitation is to be expected.

A second limitation is that the system requests that projects provide annual narrative information on the implementation and impact of individual activities. Because this represents self-reported assessments of project impact, additional information may be needed to assess whether these improvements actually occurred. Nonetheless, as part of the ongoing monitoring process, these self-reported findings can be used to help NSF staff and independent evaluators identify potentially promising practices that warrant further study. In addition, to the extent possible, these narratives will be examined during the validation process to assess whether additional information is needed to reinforce projects' claims of success. In a limited number of cases, they may also be examined through follow-up telephone interviews, currently being completed with 64 IHE participants who have been randomly selected to participate in a Customer Satisfaction Survey (NSF Generic Customer Service - OMB Clearance Number 3145-0157).

B.2.1. Statistical Methodology for Stratification and Sample Selection

This study is a census of all MSP projects and all participants within projects. No sampling methodology will be employed for respondent selection.

B.2.2. Estimation Procedure

Not Applicable

B.2.3. Degree of Accuracy Needed for the Purpose Described in the Justification

Not Applicable

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures

Not Applicable

B.2.5. Use of Periodic (Less Frequent Than Annual) Data Collection Cycles

Not Applicable

B.3. Methods for Maximizing the Response Rate and Addressing Issues of Nonresponse

Data collected for the online monitoring information system are considered part of NSF administrative requirements for awardees and therefore need to be completed in order for projects to maintain current funding levels. Therefore, we anticipate a response rate of 100 percent for each survey cycle for most of the surveys. The exceptions are the IHE Participant Survey (where we achieved a response rate of 81.4 percent of Comprehensive and Targeted MSP participants and 96 percent of Institute MSP participants for the 2005-06 collection cycle) and the Annual Survey for MSP Institute K-12 Participants (where we achieved a response rate of 78.7 percent).

Each MSP project's PI will be responsible for ensuring that individual data are obtained from partners and participants in project activities (i.e., K-12 school districts, IHE partners, and IHE participants of the Comprehensive and Targeted MSP projects and the IHE participants of the Institute MSP projects). Each PI will have access to an up-to-date list located on the Web site that will indicate who still needs to respond to the survey.

PIs will be provided with several types of assistance to ensure that they understand the importance of the data collection effort, their responsibilities for providing the data, and the technical aspects of data submission. At the start of the data collection period, a primer is provided to each MSP project outlining the online system with instructions, definitions, and guidelines for its completion. The primer also explains the purpose of the surveys to the PIs, indicates that OMB approval has been received, stresses the importance of the surveys, and provides names and contact information of NSF and contractor staff to contact with questions and comments. This primer has been successfully used in each collection cycle and revisions will be made to include the two new items (for the IHE Participant Survey and the K-12 District Survey). In addition, follow-up letters or emails are used to remind each project of the data collection. The letters are found in Attachment K and the Primer is in Attachment L.

Throughout the data collection cycle, Westat will provide respondents with online and telephone support to help them navigate the web site and address specific content issues. As the due dates for data collection approach, Westat will monitor the response patterns of individual projects to identify respondents that need additional prompting and/or assistance. Issues regarding item non-response will be handled through the Web-based system, which requires that all mandatory items be completed as a condition of final submittal to NSF.

We believe that estimate of a 100 percent response rate is consistent with the results from the previous data collection cycles (and 80 percent or higher for the IHE Participant Survey). These results are presented in Chart 10.

Chart 10. Actual Response Rates for First Data Collection Cycle and Expected Response Rates for Future Cycles
Attachment	Collection title	Response Rate for the 2005-06 collection cycle	Expected Response Rate for the 2007-2008 collection cycle
A	Annual Survey for Comprehensive and Targeted Partnership Projects	100 percent	100 percent
B	Annual IHE Participant Survey for Comprehensive and Targeted MSPs	81.4 percent	80 percent or higher
C	Annual Institution of Higher Education Survey	100 percent	100 percent
D	Annual K-12 District Survey	100 percent	100 percent
E	Annual Survey for Institute Partnership Projects	100 percent	100 percent
F	Annual IHE Participant Survey for Institute MSPs	96.0 percent	100 percent
G	Initial Survey for K-12 MSP Institute Participants	100 percent	100 percent
H	Annual Survey for MSP Institute K-12 Participants	78.7 percent	80 percent
I	Annual Survey for RETA Projects	100 percent	100 percent

 

B.4. Tests of Procedures or Methods

This data collection system has been in operation for the past years and any problems that may have arisen with system procedures have been handled during this time, providing a thorough test of the system's success. Prior to being opened to respondents, the web instruments used were extensively tested by Westat programmers and beta-testers to ensure that the programming was properly implemented; with the addition of the Institute surveys and the revisions of the remaining surveys, the system will be tested again by Westat programmers before each subsequent year of data collection begins.

B.5. Names and Telephone Numbers of Individuals Consulted

Agency Unit

James Hamos, NSF, 703-292-4687

Philis Hauser, NSF, 703-292-5104

 

Contractor

Joy Frechtling, Westat, 301-517-4006

Westat will be responsible for data collection and analysis under the direction of Gary Silverstein, 301-251-2244.

 

PIs of MSP Awards

David Pagni (714-278-2671) California State University - Fullerton (Targeted MSP awardee)

George Nelson (360-650-3637) Western Washington University (Targeted MSP awardee)

Terrence Millar (608-263-4243) University of Wisconsin - Madison (Comprehensive MSP awardee)

Judah Schwartz (617-627-6354) Tufts University (Institute MSP awardee)

5