3145-0199 2012 Part B

3145-0199

Section B

Introduction

This data collection will be a census of the universe of Targeted, Institute and Phase II projects that NSF currently funds and anticipates funding through the MSP program. Thirty-nine awards (24 Targeted and Phase II partnership awards, and 15 Institute awards) will be surveyed. Responding on behalf of these awardees during the 2011-12 school year will be an estimated 1,687 respondents who come from the following categories: PIs for partnership awards, K-12 school districts, IHE partners, and IHE participants.

B.1. Respondent Universe and Sampling Methods

At the time of this submission, the universe of Comprehensive, Targeted, Phase II and Institute projects consists of 39 MSP awards – 24 Targeted and Phase II partnership awards, and 15 Institute awards. All of these projects are being included in the monitoring system.

Similarly, all 341 IHE participants, 65 IHE partners, and all 191 K-12 school districts of Targeted and Phase II projects are included, as are all 223 IHE participants in Institute Projects and 828 K-12 participants in Institute Projects. Since the full universe for each of the populations of interest is included, no statistical sampling will be used. Chart 9 summarizes the universe and sample information.

Chart 9. Size of Universe and Sample

Survey	Population	Universe Size	Sample Size
Annual Survey for Comprehensive and Targeted Projects	PIs for Comprehensive, Targeted, and Phase II awards	24	24
Annual IHE Participant Survey for Comprehensive and Targeted MSPs	IHE participants for Comprehensive, Targeted, and Phase II MSPs	341	341
Annual Institution of Higher Education Survey	IHE partners of Comprehensive, Targeted, and Phase II MSPs	65	65
Annual K-12 District Survey	School district partners of Comprehensive, Targeted, and Phase II MSPs	191	191
Annual Survey for Institute Projects	PIs for Institute awards	15	15
Annual IHE Participant Survey for Institute MSPs	IHE participants for Institute MSPs	223	223
Initial Survey for K-12 MSP Institute Participants	K-12 Participants of Institute MSPs	267	267
Annual Survey for MSP Institute K-12 Participants	K-12 Participants of Institute MSPs	561	561
Total		1,687	1,687

 

With the exception on the IHE and K-12 participant surveys, we expect the response rate will be 100 percent. Based on our experience in previous years, we anticipate that the response rates for the IHE and K-12 participants surveys will be approximately 80 percent.

B.2. Information Collection Procedures/Limitations of the Descriptive Reports

This proposed monitoring system involves seven Web-based surveys and one paper survey, all of which have already been approved for data collection. These are the surveys for PIs of Comprehensive and Targeted partnership awards, IHE participants in Comprehensive and Targeted partnerships awards, IHE partners, and K-12 school districts. This submission requests clearance to conduct these surveys for the 2011-12 through the 2013-14 collection cycles.

NSF understands that in the absence of longitudinal data for both control and experimental groups, that it is not possible to conclusively attribute changes in student achievement (e.g., course completion, performance on accountability assessments) to the MSP program and has not and will not attribute observed changes to the program. The system is, therefore, intended to obtain descriptive information about the MSP projects and their participants in the project. Given that the MSP Management Information System is designed as a monitoring system, this limitation is to be expected. Information collected will be used as a data source for the program evaluation.

A second limitation is that the system requests that projects provide annual narrative information on the implementation and impact of individual activities. Because this represents self-reported assessments of project impact, additional information may be needed to assess whether these improvements actually occurred. Nonetheless, as part of the ongoing monitoring process, these self-reported findings can be used to help NSF staff and independent evaluators identify potentially promising practices that warrant further study. In addition, to the extent possible, these narratives will be examined during the validation process to assess whether additional information is needed to reinforce projects' claims of success. In a limited number of cases, they may also be examined through follow-up telephone interviews, currently being completed with 64 IHE participants who have been randomly selected to participate in a Customer Satisfaction Survey (NSF Generic Customer Service - OMB Clearance Number 3145-0157).

B.2.1. Statistical Methodology for Stratification and Sample Selection

This study is a census of all MSP projects and all participants within projects. No sampling methodology will be employed for respondent selection.

B.2.2. Estimation Procedure

Not Applicable

B.2.3. Degree of Accuracy Needed for the Purpose Described in the Justification

Not Applicable

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures

Not Applicable

B.2.5. Use of Periodic (Less Frequent Than Annual) Data Collection Cycles

Not Applicable

B.3. Methods for Maximizing the Response Rate and Addressing Issues of Nonresponse

Data collected for the online monitoring information system are considered part of NSF administrative requirements for awardees and therefore need to be completed in order for projects to maintain current funding levels. Therefore, we anticipate a response rate of 100 percent for each survey cycle for most of the surveys. The exceptions are the IHE Participant Survey (where we achieved a response rate of 80.9 percent of Comprehensive and Targeted MSP participants and 95.7 percent of Institute MSP participants for the 2009-10 collection cycle) and the Annual Survey for MSP Institute K-12 Participants (where we achieved a response rate of 82.8 percent).

Each MSP project's PI will be responsible for ensuring that individual data are obtained from partners and participants in project activities (i.e., K-12 school districts, IHE partners, and IHE participants of the Comprehensive and Targeted MSP projects and the IHE participants of the Institute MSP projects). Each PI will have access to an up-to-date list located on the Web site that will indicate who still needs to respond to the survey.

PIs will be provided with several types of assistance to ensure that they understand the importance of the data collection effort, their responsibilities for providing the data, and the technical aspects of data submission. At the start of the data collection period, a primer is provided to each MSP project outlining the online system with instructions, definitions, and guidelines for its completion. The primer also explains the purpose of the surveys to the PIs, indicates that OMB approval has been received, stresses the importance of the surveys, and provides names and contact information of NSF and contractor staff to contact with questions and comments. This primer has been successfully used in each collection. In addition, follow-up letters or emails are used to remind each project of the data collection. The letters are found in Attachment J and the Primer is in Attachment K.

Throughout the data collection cycle, Westat will provide respondents with online and telephone support to help them navigate the web site and address specific content issues. As the due dates for data collection approach, Westat will monitor the response patterns of individual projects to identify respondents that need additional prompting and/or assistance. Issues regarding item non-response will be handled through the Web-based system, which requires that all mandatory items be completed as a condition of final submittal to NSF.

We believe that estimate of a 100 percent response rate for the Partnership Project Surveys, IHE Partner Survey, and Initial K-12 Participant Survey, and an 85 percent or higher response rate for the IHE Participant Surveys, Annual K-12 Participant Survey, and K-12 District Survey, are consistent with the results from the previous data collection cycles. These results are presented in Chart 10.

Chart 10. Actual Response Rates for First Data Collection Cycle and Expected Response Rates for Future Cycles
Attachment	Collection title	Response Rate for the 2009-10 collection cycle	Expected Response Rate for the 2011-2012 collection cycle
A	Annual Survey for Comprehensive and Targeted Partnership Projects	100 percent	100 percent
B	Annual IHE Participant Survey for Comprehensive and Targeted MSPs	80.9 percent	85 percent or higher
C	Annual Institution of Higher Education Survey	91.3 percent	100 percent
D	Annual K-12 District Survey	82.8 percent	85 percent or higher
E	Annual Survey for Institute Partnership Projects	100 percent	100 percent
F	Annual IHE Participant Survey for Institute MSPs	95.7 percent	85 percent or higher
G	Initial Survey for K-12 MSP Institute Participants	100 percent	100 percent
H	Annual Survey for MSP Institute K-12 Participants	82.8 percent	85 percent or higher

 

 

 

B.4. Tests of Procedures or Methods

This data collection system has been in operation for the past eight years and any problems that may have arisen with system procedures have been handled during this time, providing a thorough test of the system's success. Prior to being opened to respondents, the web instruments used were extensively tested by Westat programmers and beta-testers to ensure that the programming was properly implemented; with the addition of the Institute surveys and the revisions of the remaining surveys, the system will be tested again by Westat programmers before each subsequent year of data collection begins.

B.5. Names and Telephone Numbers of Individuals Consulted

Agency Unit

James Hamos, NSF, 703-292-4687

Philis Hauser, NSF, 703-292-5104

 

Contractor

Joy Frechtling, Westat, 301-517-4006

Westat will be responsible for data collection and analysis under the direction of Gary Silverstein, 301-251-2244.

 

PIs of MSP Awards

David Pagni (714-278-2671) California State University - Fullerton (Targeted MSP awardee)

George Nelson (360-650-3637) Western Washington University (Targeted MSP awardee)

Terrence Millar (608-263-4243) University of Wisconsin - Madison (Comprehensive MSP awardee)

Judah Schwartz (617-627-6354) Tufts University (Institute MSP awardee)