REQUEST FOR CLEARANCE FOR CONDUCTING A CUSTOMER SATISFACTION STUDY FOR THE NATIONAL SCIENCE FOUNDATION’S (NSF’s) MATH AND SCIENCE PARTNERSHIP (MSP) PROGRAM
The National Science Foundation’s Math and Science Partnership (MSP) program is a major R&D effort intended to explore strategies for improving science, technology, engineering, and mathematics (STEM) education at both the K–12 and higher education levels. Building on the research evidence regarding potential factors that affect teaching and learning, the MSP has identified five key features that are common to the program’s funded partnerships. These features are as follows:
Partnership-driven—engaging disciplinary faculty in mathematics, the sciences, and engineering, as well as administrators in higher education, with key administrators, teachers, and guidance counselors in K–12 core partner organizations and other relevant entities.
Teacher quality, quantity, and diversity—enhancing the quality, quantity, and diversity of K–12 teachers of mathematics and/or the sciences. This enhancement can occur at many points along the teacher development continuum, from recruiting to teacher preparation to ongoing supports for practicing teachers.
Challenging courses and curricula—ensuring teachers and students access to challenging mathematics and/or science courses and curricula.
Evidence-based design and outcomes—building on current research on learning and teaching and contributing to the research base through the collection of evidence on learning, effectiveness of partnerships contributions made by institutions of higher education (IHE) faculty, and new institutional policies and practices.
Institutional change and sustainability—promoting sustainability of institutional change among core partners at the higher education and K–12 levels through redirection of resources and change in policies and practices.
Underlying the effort is a conjecture that the deep engagement of IHE faculty to enhance K–12 education provides a value-added beyond other approaches that share the goal of increasing the quality and equity of K–12 teaching and learning. In an effort to further diversify the types of IHEs engaged in the MSP effort, NSF established the MSP-Start Partnership component to expand the work of the program to include IHEs of varying scope, size, experience and perspectives emphasizing minority-serving institutions, community colleges and primarily undergraduate institutions.
This request for Office of Management and Budget (OMB) review asks for clearance to conduct customer satisfaction surveys with MSP-Start project participants at the end of each project’s award period. The purpose is to obtain information that can be used to understand factors that influence projects’ abilities to develop successful MSP Partnerships between IHEs and K–12 schools, especially among institutions not heavily represented in the MSP community. The proposed customer satisfaction research will meet this need by exploring MSP-Start project participants’ engagement in MSP-Start activities and the factors that facilitated or hindered each participant’s ability to contribute to the program. It will also obtain information on the extent to which project participants feel that their project successfully defined participant roles, made effective use of participants’ skills and backgrounds, respected the contributions and opinions of all participants, and instilled a shared sense of responsibility and accountability for attaining the desired goals of the MSP-Start initiative. The surveys will also gather participants’ advice for future MSP partnership efforts.
The MSP program is a major effort under the aegis of President Bush's national education initiative, No Child Left Behind. To date, NSF has made over $600 million in commitments to partnership activities in a research and development portfolio that spans the nation. The goals for the program are to
Ensure that all K-12 students have access to, are prepared for, and are encouraged to participate and succeed in challenging curricula and advanced mathematics and science courses;
Enhance the quality, quantity, and diversity of the K-12 mathematics and science teacher workforce; and
Develop evidence-based outcomes that contribute to our understanding of how students effectively learn mathematics and science.
Individual projects funded by the MSP program aim to address the aforementioned issues by incorporating a depth and quality of creative strategic actions that extend beyond commonplace approaches. Although all MSP projects share a focus on the same set of fundamental issues, individual MSP projects differ in their scope and are categorized accordingly. The MSP program provides partnership awards in the following four distinct areas:
Comprehensive Partnerships implement change in mathematics and/or science educational practices in both Institutions of Higher Education (IHEs) and in schools and school districts, resulting in improved student achievement across the K-12 continuum.
Targeted Partnerships focus on improved K-12 student achievement in a narrower grade range or disciplinary focus within mathematics or science.
Institute Partnerships, also referred to as Teacher Institutes for the 21st Century, focus on the development of mathematics and science teachers as school- and district-based intellectual leaders and master teachers.
MSP-Start Partnerships are for awardees new to the MSP program, especially from minority-serving institutions, community colleges and primarily undergraduate institutions, to support the necessary data analysis, project design, evaluation and team building activities needed to develop a full MSP Targeted or Institute Project.
This clearance request covers a series of online surveys that are designed to obtain customer satisfaction data from principal investigators (PIs) and other participants from each of the MSP-Start Partnership projects.
MSP-Start Partnerships are a new component of the MSP Program, and currently no data are being collected from these awardees. In an MSP-Start Partnership award, a lead organization identifies the appropriate K–12, higher education, and other institutions that are anticipated to come together to form a partnership, and engages a broad team of personnel in these organization in one to two years of partnership activities that are a necessary prerequisite to a strong project design. MSP-Start awardees conduct the data collection, analysis, team building, and evaluation necessary for developing a proposal for a full MSP Targeted or Institute Partnership. In order to increase engagement in the MSP program and enhance the quality of MSP partnerships, it is essential that NSF understand the motivations and experiences of MSP-Start participants, as well as the best practices for developing successful MSP partnerships, especially among IHEs that are not heavily represented among the MSP community, and share this information with current and future projects.
The information to be collected in these online surveys represents the minimum effort required to evaluate customer satisfaction with the MSP-Start program. It is expected that the surveys will take 45 minutes for each respondent. Individual respondents will not be asked to participate in more than one survey and the survey will only be administered once to each participant.
Westat, NSF's contractor for the online management information system, has provided NSF with a series of reports displaying aggregated data for MSP Comprehensive, Targeted, and Institute projects. While these reports have provided summary data on project participation trends from full partnership projects, they have not been able to provide data on the factors that contribute to the development of successful partnerships. Now that NSF has launched the MSP-Start program which supports one to two years of partnership activities focused on building a strong MSP project design, the Foundation is in a position to collect and report on participants’ satisfaction with their experiences associated with planning and developing MSP partnerships. The report that results from these surveys will be used to disseminate information to current and future MSP projects about potentially promising practices that lead to strong partnerships prepared to engage in MSP-type work.
The MSP-Start data collection effort will make maximum use of computer technology to minimize the response burden. Participants will use a worldwide Web browser to submit the required data over the Internet using specially developed MSP online data collection system software. EHR favors Web-based systems because they facilitate respondents' data entry across computer platforms. One feature of the system is the thorough editing of data for completeness, validity, and consistency prior to final submittal. Editing is performed as data are entered. Questionable or incomplete entries are called to respondents' attention before they are submitted to NSF. Features such as automatic tabulations, checkboxes, standard menus, and predefined charts and graphics facilitate the reporting process, provide useful and rapid feedback to the data providers, and reduce burden.
The information collected via these surveys will not be duplicated by other data collection. An internal panel of program staff have reviewed the surveys to verify that the information sought is not already available and that the surveys are congruent with the agency’s customer satisfaction program and NSF’s need to obtain information about MSP-Start project activities.
No information is to be collected from small businesses.
Without collecting this information, it would not be possible to systematically examine best practices for attracting motivated PIs to participate with the MSP, and effectively using the skills and knowledge of IHE faculty, K–12 district and school representatives, and outside business organizations to create successful partnerships. In addition, this information can inform future NSF efforts that will enhance the satisfaction of participants in other programs beyond the MSP.
A.7. Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6
Not applicable.
Westat shared drafts of the MSP-Start participant surveys with MSP-Start project PIs during the MSP Program’s Learning Network Conference in January 2009 and January 2010. Feedback from the project PIs has been incorporated into the final drafts of the surveys.
Not applicable.
Respondents will be advised that no information on specific individuals will be reported and no quotes or information that could reveal the identity or affiliation of individual respondents will be reported to NSF.
The surveys will include questions on participants’ perception of their MSP-Start program’s success in building a successful partnership, including whether or not their skills and backgrounds were maximized by their MSP-Start project. As such, respondents may be concerned about their responses impacting their employment, salary, promotion, or project participation. Respondents will be assured that their individual responses will not be shared with others within their organization, with their MSP project, or with NSF program staff, and that all results will only be presented in an anonymous and aggregated form.
The estimated response burden for each individual respondent is approximately 45 minutes.
All respondents will be MSP-Start project participants. Each respondent will complete a single survey. The survey will be administered once at the end of each MSP-Start award and the burden will be approximately 45 minutes per survey—with an additional 30 minutes for PIs to complete an administrative sign-in sheet (see Chart 1). A total of 126 respondents from the first cohort of MSP-Start projects will be surveyed in 2010, 112 respondents from the second cohort of MSP-Start projects will be surveyed in 2011, and 28 respondents from the third cohort will be surveyed in 2012. Overall, 266 respondents will complete a survey, for a total response burden of 209 hours over a 3-year period.
Chart 1. Burden Hours for MSP-Start Surveys
Participant |
Number of Respondents |
Burden per Respondent |
Total Burden (hours)* |
Cohort 1 |
|||
PI |
9 |
75 minutes |
11.25 |
Co-PI |
18 |
45 minutes |
13.50 |
IHE Participants |
45 |
45 minutes |
33.75 |
K–12 Participants |
36 |
45 minutes |
27.00 |
Business Participants |
18 |
45 minutes |
13.50 |
Total |
126 |
|
99.00 |
Cohort 2 |
|||
PI |
8 |
75 minutes |
10.00 |
Co-PI |
16 |
45 minutes |
12.00 |
IHE Participants |
40 |
45 minutes |
30.00 |
K–12 Participants |
32 |
45 minutes |
24.00 |
Business Participants |
16 |
45 minutes |
12.00 |
Total |
112 |
|
88.00 |
Cohort 3 |
|||
PI |
2 |
75 minutes |
2.50 |
Co-PI |
4 |
45 minutes |
3.00 |
IHE Participants |
10 |
45 minutes |
7.50 |
K–12 Participants |
8 |
45 minutes |
6.00 |
Business Participants |
4 |
45 minutes |
3.00 |
Total |
28 |
|
22.00 |
Total |
|||
PI |
19 |
75 minutes |
23.75 |
Co-PI |
38 |
45 minutes |
28.50 |
IHE Participants |
95 |
45 minutes |
71.25 |
K–12 Participants |
76 |
45 minutes |
57.00 |
Business Participants |
38 |
45 minutes |
28.50 |
Total |
266 |
|
209.00 |
*=Number of Respondents x Burden Per Respondent/60 minutes |
|||
The customer satisfaction surveys will utilize four distinct surveys—one for MSP-Start project PIs, one for IHE Participants, one for K–12 Participants, and one for Business Participants. The response burden for each of these surveys is 45 minutes. The 75-minute burden for the PI includes time required to complete the administrative screens that are needed to initiate the surveys for other members of the MSP-START partnership.
Chart 2. OMB Burden Calculation for Customer Satisfaction Survey, by Survey Form
Participant |
Number of Respondents |
Burden per Respondent |
Total Burden (hours)* |
Cohort 1 |
|||
PI administrative form |
9 |
30 minutes |
4.50 |
PI/Co-PI |
27 |
45 minutes |
20.25 |
IHE Participants |
45 |
45 minutes |
33.75 |
K–12 Participants |
36 |
45 minutes |
27.00 |
Business Participants |
18 |
45 minutes |
13.50 |
Total |
|
|
99.00 |
Cohort 2 |
|||
PI administrative form |
8 |
30 minutes |
4.00 |
PI/Co-PI |
24 |
45 minutes |
18.00 |
IHE Participants |
40 |
45 minutes |
30.00 |
K–12 Participants |
32 |
45 minutes |
24.00 |
Business Participants |
16 |
45 minutes |
12.00 |
Total |
|
|
88.00 |
Cohort 3 |
|||
PI Administrative form |
2 |
30 minutes |
1.00 |
PI/Co-PI |
6 |
45 minutes |
4.50 |
IHE Participants |
10 |
45 minutes |
7.50 |
K–12 Participants |
8 |
45 minutes |
6.00 |
Business Participants |
4 |
45 minutes |
3.00 |
Total |
|
|
22.00 |
Total |
|||
PI administrative form |
19 |
30 minutes |
9.50 |
PI/Co-PI |
57 |
45 minutes |
42.75 |
IHE Participants |
95 |
45 minutes |
71.25 |
K–12 Participants |
76 |
45 minutes |
57.00 |
Business Participants |
38 |
45 minutes |
28.50 |
Total |
|
|
209.00 |
*=Number of Respondents x Burden Per Respondent/60 minutes |
|||
The overall cost to respondents over the 3-year period is estimated to be $8,185. The hourly wage rates for CoPI respondents and IHE Participant respondents were based on information found in the Department of Education's National Center for Educational Statistics Employees in Postsecondary Institutions, Fall 2008, and Salaries of Full-Time Instructional Staff, 2008-09 (Table 4, Adjusted 9-month average salaries of full-time instructional staff at Title IV degree-granting institutions, by academic rank, control and level of institution, and gender: United States, academic year 2009-09), http://nces.ed.gov/pubs2010/2010165.pdf. The hourly wage rates for Business, and PI and K-12 Participant respondents were based on information found in the U.S. Department of Labor, Bureau of Labor Statistics, Occupational Employment Statistics (OES) Survey, May 2008 National Occupational Employment and Wage Estimates, http://www.bls.gov/oes/current/oes_nat.htm.
Calculations are shown in Chart 3.
Chart 3. Annualized Cost to Respondents, by Type of Respondent
|
Hour Burden |
Estimated Hourly Wage Rate |
Estimated Cost to Respondents* |
PI (IHE Faculty) |
9.5 |
$ 48.87 |
$ 464.27 |
CoPI (IHE Faculty) |
42.75 |
$48.87 |
$ 2089.19 |
IHE Participants |
71.25 |
$35.21 |
$2,508.71 |
K–12 Participants |
57.00 |
$33.48 |
$ 1908.36 |
Business Participants |
28.50 |
$42.60 |
$ 1214.10 |
Total |
209.00 |
|
$8,184.63 |
* Annual Average Hour Burden x Estimated Hourly Wage Rate
There is no overall annual cost burden to respondents or record keepers that results from the MSP program other than the time spent responding to the surveys that are attached as appendices to this request.
The total estimated cost to the government of all data collection, analysis, and reporting activities for this survey is $87,766. The total estimated costs are shown in Chart 4.
Chart 4. Estimated Costs to the Federal Government of Collection
Personnel |
$29,500 |
Other Direct Costs |
$1,992 |
Overhead |
$31,919 |
G&A and Fee |
$24,355 |
Total Costs |
$87,766 |
Not applicable.
Chart 5 summarizes the timeline for the data collection and annual contractor reports for the first year of data collection.
Chart 5. MSP Work plan
Data Collection Tasks |
Date to start |
Date to complete |
Programming of data collection software |
12/2009 |
2/2010 |
Software testing and revision |
2/2010 |
3/2010 |
Submission of clearance package to OMB |
|
|
Receipt of OMB clearance (assuming received on day 60) |
|
|
Web data collection |
3/2010 |
4/2010 |
Follow-up Phone Calls to PIs |
5/2010 |
5/2010 |
Final Cleaning and Validation of Databases |
6/2010 |
6/2010 |
Analysis and Tabulations for Report |
7/2010 |
8/2010 |
Report to NSF |
|
9/2010 |
Like many agencies, NSF is reducing its reliance on formal (i.e., traditional) publication methods and publication formats. Westat is conducting this third-party study of the MSP program on behalf of NSF and is forbidden contractually from publishing results unless NSF has made a specific exception. In short, all products of the collections are the property of NSF. After the products are delivered, NSF determines whether the quality of the products deserves publication verbatim by NSF, i.e., NSF is the exclusive publisher of the information being gathered. Often it is only after seeing the quality of the information delivered by the study that NSF decides the format (raw or analytical) and manner (in the NSF-numbered product Online Document System (ODS) or simply a page on the NSF Web site) in which to publish.
Before the conclusion of the study, both NSF and the funded MSP projects may use preliminary data to improve management and performance. For example, data generated by this study are expected to be inputs to other internal and external NSF reports (e.g., the GPRA Annual Performance Plan). At this time, NSF has no set timeline for publishing interim reports from this study.
Not applicable.
None.
This data collection will be a census of the universe of MSP-Start projects that NSF currently funds and anticipates funding through the MSP program. Nine projects will be surveyed in 2010, and eight projects will be surveyed in 2011. In 2010, responding on behalf of these awardees will be an estimated 125 respondents who come from the following categories: PIs and CoPIs for MSP-Start awards, K-12 district participants, IHE participants, and profit/not-for-profit participants.
At the time of this submission, the universe of NSF projects consists of nine MSP-Start awards ending in 2010, eight MSP-Start awards ending in 2011, and two MSP-START awards ending in 2012. All of these projects are being included in the MSP-Start surveys.
Similarly, all project PIs and CoPIs, IHE participants, K-12 district participants, and other business participants are included. Since the full universe for each of the populations of interest is included, no statistical sampling will be used. Chart 6 summarizes the universe and sample information.
Chart 6. Size of Universe and Sample
Survey |
Population |
Universe Size |
Sample
|
Survey for MSP-Start Principal Investigators |
PIs and CoPIs for MSP-Start projects |
57 |
57 |
Survey for MSP-Start IHE Participants |
IHE participants in MSP-Start projects |
95 |
95 |
Survey for MSP-Start K–12 Participants |
K–12 participants in MSP-Start projects |
76 |
76 |
Survey for MSP-Start Profit/Not-For-Profit Participants |
Profit/Not-For-Profit Business participants in MSP-Start projects |
38 |
38 |
Total |
266 |
266 |
|
We expect the response rate will be 100 percent.
The proposed survey system involves four Web-based surveys. These are the surveys for PIs of MSP-Start awards, IHE participants, K-12 participants, and other profit/not-for-profit business partner participants. Because Westat does not have names or contact information for MSP-Start project participants other than the PIs, the PI of each MSP project (who is required to maintain such contact information) will be asked to register all project participants to complete the survey using an Administrative screen on the Survey for MSP-Start Principal Investigators. The PI will also be responsible for contacting all respondents and obtaining their cooperation with the survey.
This study is a census of all MSP-Start projects and all participants within projects. No sampling methodology will be employed for respondent selection.
Not Applicable
Not Applicable
Not Applicable
Not Applicable
Data collected for the online monitoring information system are considered part of NSF administrative requirements for awardees. As the premise of the MSP-Start program is to support the necessary partnership activities needed to develop a full MSP Targeted or Institute Partnership, NSF anticipates that MSP-Start awardees will submit proposals for future funding from the MSP program. As such, we anticipate the full cooperation of MSP-Start participants and a response rate of 100 percent for each survey.
Each MSP project's PI will be responsible for ensuring that individual data are obtained from participants in project activities. Each PI will have access to an up-to-date list located on the Web site that will indicate who still needs to respond to the survey. In addition, throughout the data collection cycle, Westat will provide respondents with online and telephone support to help them navigate the web site and address specific content issues. As the due dates for data collection approach, Westat will monitor the response patterns of individual projects to identify respondents that need additional prompting and/or assistance. Issues regarding item non-response will be handled through the Web-based system, which requires that all mandatory items be completed as a condition of final submittal to NSF.
Prior to being opened to respondents, the web instruments will be extensively tested by Westat programmers and beta-testers to ensure that the programming is properly implemented. The system will be tested again by Westat programmers before each subsequent year of data collection begins.
Agency Unit
Philis Hauser, NSF, 703-292-5104
Contractor
Joy Frechtling, Westat, 301-517-4006
ATTACHMENTS TO SECTION B: Draft surveys for PIs, IHE Participants, K–12 Participants, and Profit/Not-For-Profit Organization Participants
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Statement () |
| Author | Gavin Fulmer |
| File Modified | 0000-00-00 |
| File Created | 2021-02-04 |