3145-0199 Part B 2015

Section B

Introduction

This data collection will be a census of the universe of Targeted, Institute and Phase II projects that NSF currently funds and anticipates funding through the MSP program. Forty-nine awards (37 Targeted and Phase II partnership awards and 12 Institute awards) will be surveyed. Responding on behalf of these awardees during the 2013–14 school year will be an estimated 1,936 respondents who come from the following categories: PIs for partnership awards, K-12 school districts, IHE partners, and IHE participants.

B.1. Respondent Universe and Sampling Methods

At the time of this submission, the universe of Comprehensive, Targeted, Phase II and Institute projects consists of 49 MSP awards – 37 Targeted and Phase II partnership awards, and 12 Institute awards. All of these projects are being included in the monitoring system.

Similarly, all 386 IHE participants, 84 IHE partners, and all 277 K-12 school districts of Targeted and Phase II projects are included, as are all 116 IHE participants in Institute Projects and 947 K-12 participants in Institute Projects. Since the full universe for each of the populations of interest is included, no statistical sampling will be used. Chart 8 summarizes the universe and sample information.

Chart 8.—Size of Universe and Sample

Survey	Population	Universe Size	Sample Size
Annual Survey for Comprehensive and Targeted Projects	PIs for Comprehensive, Targeted, and Phase II awards	37	37
Annual IHE Participant Survey for Comprehensive and Targeted MSPs	IHE participants for Comprehensive, Targeted, and Phase II MSPs	386	386
Annual Institution of Higher Education Survey	IHE partners of Comprehensive, Targeted, and Phase II MSPs	84	84
Annual K-12 District Survey	School district partners of Comprehensive, Targeted, and Phase II MSPs	277	277
Annual Survey for Institute Projects	PIs for Institute awards	12	12
Annual IHE Participant Survey for Institute MSPs	IHE participants for Institute MSPs	116	116
Initial Survey for K-12 MSP Institute Participants	K-12 Participants of Institute MSPs	77	77
Annual Survey for MSP Institute K-12 Participants	K-12 Participants of Institute MSPs	947	947
Total		1,936	1,936

 

With the exception on the IHE and K-12 participant surveys, we expect the response rate will be 100 percent. Based on our experience in previous years, we anticipate that the response rates for the IHE and K-12 participant surveys will be approximately 80 percent.

B.2. Information Collection Procedures/Limitations of the Descriptive Reports

This proposed monitoring system involves eight Web-based surveys, all of which have already been approved for data collection. These are the surveys for PIs of Comprehensive and Targeted partnership awards, IHE participants in Comprehensive and Targeted partnerships awards, IHE partners, and K-12 school districts. This submission requests clearance to conduct these surveys for the 2015–16 through the 2018–19 collection cycles.

NSF understands that in the absence of longitudinal data for both control and experimental groups, that it is not possible to conclusively attribute changes in student achievement (e.g., course completion, performance on accountability assessments) to the MSP program and has not and will not attribute observed changes to the program. The system is, therefore, intended to obtain descriptive information about the MSP projects and their participants in the project. Given that the MSP Management Information System is designed as a monitoring system, this limitation is to be expected. Information collected will be used as a data source for the program evaluation.

A second limitation is that the system requests that projects provide annual narrative information on the implementation and impact of individual activities. Because this represents self-reported assessments of project impact, additional information may be needed to assess whether these improvements actually occurred. Nonetheless, as part of the ongoing monitoring process, these self-reported findings can be used to help NSF staff and independent evaluators identify potentially promising practices that warrant further study. In addition, to the extent possible, these narratives will be examined during the validation process to assess whether additional information is needed to reinforce projects’ claims of success.

B.2.1. Statistical Methodology for Stratification and Sample Selection

This study is a census of all MSP projects and all participants within projects. No sampling methodology will be employed for respondent selection.

B.2.2. Estimation Procedure

Not Applicable

B.2.3. Degree of Accuracy Needed for the Purpose Described in the Justification

Not Applicable

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures

Not Applicable

B.2.5. Use of Periodic (Less Frequent Than Annual) Data Collection Cycles

Not Applicable

B.3. Methods for Maximizing the Response Rate and Addressing Issues of Nonresponse

Data collected for the online monitoring information system are considered part of NSF administrative requirements for awardees and therefore need to be completed in order for projects to maintain current funding levels. Therefore, we anticipate a response rate of 100 percent for each survey cycle for most of the surveys. The exceptions are the IHE Participant Survey (where we achieved a response rate of 83.4percent of Comprehensive and Targeted MSP participants and 84.5 percent of Institute MSP participants for the 2013–14 collection cycle) and the Annual Survey for MSP Institute K-12 Participants (where we achieved a response rate of 83.4 percent).

Each MSP project’s PI will be responsible for ensuring that individual data are obtained from partners and participants in project activities (i.e., K-12 school districts, IHE partners, and IHE participants of the Comprehensive and Targeted MSP projects and the IHE participants of the Institute MSP projects). Each PI will have access to an up-to-date list located on the Web site that will indicate who still needs to respond to the survey.

PIs will be provided with several types of assistance to ensure that they understand the importance of the data collection effort, their responsibilities for providing the data, and the technical aspects of data submission. For example, throughout the data collection cycle, Westat will provide respondents with online and telephone support to help them navigate the web site and address specific content issues. The “Home” screen of each survey provides names and contact information of contractor staff to contact with questions and comments. Text throughout the online surveys provides screen-specific instructions, definitions, and guidelines for survey completion. A “Help” screen, accessible from any point in the surveys, provides navigational instructions and answers frequently asked questions. As the due dates for data collection approach, Westat will monitor the response patterns of individual projects to identify respondents that need additional prompting and/or assistance. Issues regarding item non-response will be handled through the Web-based system, which requires that all mandatory items be completed as a condition of final submittal to NSF. In addition, follow-up letters or emails are used to remind each project of the data collection. The letters are found in Attachment J.

We believe that estimates of a 100 percent response rate for the Partnership Project Surveys, IHE Partner Survey, and Initial K-12 Participant Survey, and an 80 percent or higher response rate for the IHE Participant Surveys, Annual K-12 Participant Survey, and K-12 District Survey, are consistent with the results from the previous data collection cycles. These results are presented in Chart 9.

Chart 9.—Actual Response Rates for First Data Collection Cycle and Expected Response Rates for Future Cycles

Attachment	Collection title	Response Rate for 2013–14 collection cycle	Expected Response Rate for 2015–16 collection cycle
A	Annual Survey for Comprehensive and Targeted Partnership Projects	100 percent	100 percent
B	Annual IHE Participant Survey for Comprehensive and Targeted MSPs	83.4 percent	80 percent or higher
C	Annual Institution of Higher Education Survey	90.5 percent	100 percent
D	Annual K-12 District Survey	83.4 percent	80 percent or higher
E	Annual Survey for Institute Partnership Projects	91.7 percent	100 percent
F	Annual IHE Participant Survey for Institute MSPs	84.5 percent	80 percent or higher
G	Initial Survey for K-12 MSP Institute Participants	42.9 percent	100 percent
H	Annual Survey for MSP Institute K-12 Participants	66.6 percent	80 percent or higher

 

B.4. Tests of Procedures or Methods

This data collection system has been in operation for the past 11 years and any problems that may have arisen with system procedures have been handled during this time, providing a thorough test of the system’s success. Prior to being opened to respondents, the web instruments used were extensively tested by Westat programmers and beta-testers to ensure that the programming was properly implemented; the system will be tested again by Westat programmers before each subsequent year of data collection begins.

B.5. Names and Telephone Numbers of Individuals Consulted

Agency Unit

Kathleen Bergin, (703) 292-5171

Philis Hauser, NSF, 703-292-5104

 

Contractor

Joy Frechtling, Westat, 301-517-4006

Westat will be responsible for data collection and analysis under the direction of Gary Silverstein,
301-251-2244.

6