Supporting Statement for Math & Science Partnerships Customer Satisfaction Survey

Supporting Statement

REQUEST FOR CLEARANCE FOR CONDUCTING A CUSTOMER SATISFACTION STUDY FOR THE NATIONAL SCIENCE FOUNDATION’S (NSF’s) MATH AND SCIENCE PARTNERSHIP (MSP) PROGRAM

Section A

Introduction

The National Science Foundation’s Math and Science Partnership (MSP) program is a major R&D effort intended to explore strategies for improving science, technology, engineering, and mathematics (STEM) education at both the K–12 and higher education levels. Building on the research evidence regarding potential factors that affect teaching and learning, the MSP has identified five key features that are common to the program’s funded partnerships. These features are as follows:

• Partnership-driven—engaging disciplinary faculty in mathematics, the sciences, and engineering, as well as administrators in higher education, with key administrators, teachers, and guidance counselors in K–12 core partner organizations and other relevant entities.

• Teacher quality, quantity, and diversity—enhancing the quality, quantity, and diversity of K–12 teachers of mathematics and/or the sciences. This enhancement can occur at many points along the teacher development continuum, from recruiting to teacher preparation to ongoing supports for practicing teachers.

• Challenging courses and curricula—ensuring teachers and students access to challenging mathematics and/or science courses and curricula.

• Evidence-based design and outcomes—building on current research on learning and teaching and contributing to the research base through the collection of evidence on learning, effectiveness of partnerships contributions made by institutions of higher education (IHE) faculty, and new institutional policies and practices.

• Institutional change and sustainability—promoting sustainability of institutional change among core partners at the higher education and K–12 levels through redirection of resources and change in policies and practices.

Underlying the effort is a conjecture that the deep engagement of STEM faculty to enhance K-12 education provides a value-added beyond other approaches that share the goal of increasing the quality and equity of K–12 teaching and learning. Central is the belief that the rich content knowledge that disciplinary faculty bring can improve what is learned by teachers through professional development, as well as influence the K–12 curriculum. In addition, the MSP posits that such engagement should and will benefit the partner IHEs as well as those at the K–12 level. While programmatic outcomes extensively are targeted at the K-12 level, the MSP program supports the transformation of STEM education across the K–20 educational spectrum.

Analyses of data obtained through the MSP Management Information System (MIS) (OMB # 3145-0136) reveal that 30 percent of STEM faculty have discontinued their participation with the MSP program after at least one year of involvement. Furthermore, findings show much variation in the percentage of these “leavers” among projects funded under the MSP program. While some projects had less than 10 percent of STEM faculty leave, other projects had well over 30 percent discontinue their participation after one or two years.

What are the best practices for recruiting and sustaining STEM faculty engagement with the MSP program? Though the MSP MIS provides some basic information about the characteristics of “stayers” and “leavers,” the available data is insufficient to answer this question. The decision to continue or discontinue participation may be influenced by a range of individual and project-specific factors. For example, leavers may become disinterested or dissatisfied with their MSP experience, may have held a finite and time-limited role in MSP, or may have changed job locations for reasons unrelated to the MSP. The decision to continue or discontinue may also be influenced by the type and level of support provided to STEM faculty (e.g., recognition, financial support). Understanding the project actions and individual factors that facilitate or hinder faculty participation in MSP would help NSF disseminate information to current and future projects that are looking to enhance their STEM faculty recruitment and retention efforts.

This request for Office of Management and Budget (OMB) review asks for clearance to conduct customer satisfaction interviews with a sample of the 1,545 STEM faculty who have participated in the MSP program since its inception in 2002. The purpose is to obtain information that can be used to understand factors that may affect individuals’ decisions to continue or discontinue their participation in MSP. The proposed customer satisfaction research will meet this need by exploring differences in STEM faculty satisfaction and engagement with the MSP program for stayers and leavers. It will also obtain information about the incidence and efficacy of specific policies and practices (both project-specific and IHE-specific) that are being used to recruit and retain STEM faculty in MSP-supported teaching and research efforts. The focus on STEM faculty (as opposed to all faculty participants) reflects the emphasis that the NSF MSP places on the role of disciplinary faculty in these projects.

Timely clearance of this request is critical in order for NSF to meet the Congressional mandate in P. L. 107-368, Sections 9 and 19, to evaluate the MSP program and provide Congress with ongoing results from this evaluation and for NSF to meet its Government Performance and Results Act (GPRA) and OMB's Program Assessment Rating Tool (PART) accountability requirements. It will also enable the NSF to disseminate information in a timely manner regarding practices that projects are using to recruit and retain STEM faculty.

A. Overview of the MSP Program

The MSP program is a major effort under the aegis of President Bush's national education initiative, No Child Left Behind. To date, NSF has made over $600 million in commitments to partnership activities in a research and development portfolio that spans the nation. The goals for the program are to

• Ensure that all K-12 students have access to, are prepared for, and are encouraged to participate and succeed in challenging curricula and advanced mathematics and science courses;

• Enhance the quality, quantity, and diversity of the K-12 mathematics and science teacher workforce; and

• Develop evidence-based outcomes that contribute to our understanding of how students effectively learn mathematics and science.

Individual projects funded by the MSP program aim to address the aforementioned issues by incorporating a depth and quality of creative strategic actions that extend beyond commonplace approaches. Although all MSP projects share a focus on the same set of fundamental issues, individual MSP projects differ in their scope and are categorized accordingly. The MSP program provides partnership awards in the following three distinct areas:

• Comprehensive Partnerships implement change in mathematics and/or science educational practices in both Institutions of Higher Education (IHEs) and in schools and school districts, resulting in improved student achievement across the K-12 continuum.

• Targeted Partnerships focus on improved K-12 student achievement in a narrower grade range or disciplinary focus within mathematics or science.

• Institute Partnerships, also referred to as Teacher Institutes for the 21st Century, focus on the development of mathematics and science teachers as school- and district-based intellectual leaders and master teachers.

The first cohort of MSP projects includes Comprehensive partnerships and Targeted partnerships. The second cohort consists of Comprehensive partnerships and Targeted partnerships, as well as one prototype Institute partnership award. The third cohort consists of Institute partnerships.

A.1. Circumstances Requiring the Collection of Data

As described above, an analysis of data from the MSP MIS reveals that 30 percent of STEM faculty have discontinued their participation with the MSP program over time, and indicate variation in the proportion of STEM faculty leaving individual MSP projects. This presents a potential challenge to the program, since the long-term engagement of STEM faculty has the potential to be an important component of the program’s approach for enhancing K-20 education. It is essential that the Foundation understand the best practices for recruiting and sustaining STEM faculty for the MSP program, and share this information with current and future projects.

The information to be collected in these telephone interviews represents the minimum effort required to evaluate customer satisfaction with their involvement in teaching and research through the MSP program. It is expected that the telephone protocols will be refined to take less than one hour for each respondent. Individual respondents will not be asked to participate in more than one telephone interview and the survey will only be administered once. The total number and schedule of interviews will be monitored and approved by the Office of the Director.

A.2. Purposes and Uses of the Data

The purpose of this effort is to evaluate customer satisfaction of STEM faculty participating in teaching or research through the MSP program, and to determine the efficacy of policies put in place to increase such satisfaction.

Westat, NSF's contractor for the online management information system, has provided NSF with a series of reports displaying annual aggregated data for all MSP projects, as well as project-specific reports for each MSP project. While these reports have provided summary data on broad participation trends, they have not been able to delve more deeply into the factors that influence these trends. The report that results from this survey will be used to disseminate information to current and future MSP projects about potentially promising practices that are designed to support participating STEM faculty and promote their long-term participation in the MSP program.

A.3. Use of Information Technology to Reduce Burden

Although the MSP MIS is administered via an online survey, the exploratory nature of the questions that are to be addressed will require the use of open-ended interviews with a small number of STEM faculty. As such, the proposed interviews will be conducted via telephone.

A.4. Efforts to Identify Duplication

The information collected via these interviews will not be duplicated by other data collection. Interviewers will have access to existent responses to data for individual respondents (obtained through the MSP MIS), thereby eliminating the need for repetitive questions and reducing the length of an interview. An internal panel of program staff have reviewed the proposed interview protocols to verify that the information sought is not already available and that the interviews are congruent with the agency’s customer satisfaction program and the Foundation’s need to obtain information about practices designed to maximize participation among STEM faculty.

A.5. Small Business

No information is to be collected from small businesses.

A.6. Consequences of Not Collecting the Information

It is essential for the success of the MSP program that customer satisfaction data of STEM faculty participating with the MSP program be collected. Without collecting this information, it would not be possible to systematically examine “best practices” for attracting motivated STEM faculty to participate with the MSP, nor ways to sustain their involvement with the program. In addition, this information can inform future NSF efforts that will enhance the satisfaction of participating STEM faculty in other programs beyond the MSP.

A.7. Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6

Not applicable.

A.8. Consultation Outside the Agency

Not applicable.

A.9. Payments or Gifts to Respondents

Not applicable.

A.10. Assurance of Confidentiality

Respondents will be advised that no information on specific individuals will be reported and no quotes or information that could reveal the identity or affiliation of individual respondents will be reported to NSF.

A.11. Questions of a Sensitive Nature

The interviews will include questions on participants’ decision to continue or discontinue involvement with the MSP program. As such, responses may potentially address individuals’ concerns of employment, tenure promotion, or project personnel. Respondents will be assured that their responses will not be shared with other individuals at their IHE, with their MSP project, or with NSF program staff, and that all results will only be presented in an anonymous and aggregated form.

A.12. Estimates of Response Burden

The estimated response burden for each individual respondent is approximately one hour.

A.12.1. Number of Respondents, Frequency of Response, and Annual Hour Burden

A total of 64 respondents will be interviewed. All respondents will be current and former STEM faculty participants who have participated in the MSP program. Each respondent will complete a single interview. The annual burden will be approximately 60 minutes (see Chart 1).

Chart 1. Annual Burden Hours for IHE Faculty Telephone Interviews

Participant	Number of Respondents	Burden per Respondent	Total Annual Burden*
IHE Faculty	64	1 hour	64 hours
Total Respondents	64		64 hours
*=Number of Respondents x Burden Hours per Response

A.12.2. Hour Burden Estimates by Each Form and Aggregate Hour Burdens

The customer satisfaction interviews will utilize three distinct protocols—one for IHE STEM faculty who have remained with their MSP over time, one for IHE STEM faculty who have discontinued their participation in MSP after one or more years, and another for IHE STEM faculty who acquired tenure status during their MSP participation. The response burden for each of these interviews is 60 minutes.

Chart 2. OMB Burden Calculation for Customer Satisfaction Survey, by Survey Form 

Participant	Number of Respondents	Burden per Respondent	Total Annual Burden*
IHE STEM faculty who remain with the MSP program	28	1 hour	28 hours
IHE STEM faculty who leave the MSP program	28	1 hour	28 hours
IHE STEM faculty who gained tenure status during MSP participation	8	1 hour	8 hours
Total Respondents	64		64 hours
*=Number of Respondents x Burden Hours per Response

A.12.3. Estimates of Annualized Cost to Respondents for the Hour Burdens

The overall annual costs to respondents for burden hours are estimated to be $2,787. The hourly wage rates were based on information found in the Department of Education's National Center for Educational Statistics Employees in Postsecondary Institutions, Fall 2005 and Salaries of Full-Time Instructional Faculty, 2005-06 (Table 5, Average salaries of full-time instructional faculty at Title IV degree-granting institutions, by academic rank, control and level of institution, and gender: United States, academic year 2005-06), http://nces.ed.gov/pubs2007/2007150.pdf). Calculations are shown in Chart 3.

Chart 3. Annualized Cost to Respondents, by Type of Respondent                                    

	Annual Average Hour Burden	Estimated Hourly Wage Rate	Estimated Cost to Respondents*
IHE STEM faculty	64	43.55	$2,787

* Annual Average Hour Burden  x  Estimated Hourly Wage Rate

A.13. Estimate of Total Capital and Startup Costs/Operation and Maintenance Costs to Respondents or Record Keepers

There is no overall annual cost burden to respondents or record keepers that results from the MSP program other than the time spent responding to the MSP MIS and the interview protocols that are attached as appendices to this request.

A.14. Estimates of Costs to the Federal Government

The total estimated cost to the government of all data collection, analysis, and reporting activities for this survey is $23,691. The total estimated costs are shown in Chart 4.

Chart 4. Estimated Costs to the Federal Government of Collection

Personnel	$8,274
Other Direct Costs
Support Services	0
Travel	0
Communication & Supplies	$2,019
Indirect Costs
Overhead	$8,960
G&A and Fee	$4,438
Total Costs	$23,691

A.15. Changes in Burden

Not applicable.

A.16. Plans for Publication, Analysis, and Schedule

The results of this study will be prepared as a technical report for the NSF, which will use the findings to disseminate information to projects that are conducting similar K-20 programs that involve the expertise and support of STEM faculty. The schedule is shown below.

Begin telephone interviews March 1, 2008

Complete telephone interviews May 1, 2008

Submit summary report of findings July 2008

A.17. Approval to Not Display Expiration Date

Not applicable.

A.18. Exceptions to Item 19 or OMB Form 83-I

None.

Section B

Introduction

B.1. Respondent Universe and Sampling Methods

As previously described, a key purpose of the MSP program is to engage STEM faculty in teaching and research on K-20 disciplinary education efforts. Therefore, the criteria used to select the 64 faculty who will participate in the customer satisfaction interviews will be designed to reflect program priorities. Specifically:

• 40 respondents will be tenured/tenure track STEM faculty from Comprehensive and Targeted projects who are based in a doctoral/research or research university.

• 16 respondents will be STEM faculty from Institute projects. Because of the small number of Institute faculty, we are not proposing to limit these interviews to participants in doctoral/research or research universities.

• 8 respondents will be STEM faculty from Comprehensive and Targeted projects who acquired tenure during the time they were participating in the MSP. These interviews will be designed to explore whether or not their MSP participation affected their acquisition of tenure.

Chart 5 provides a breakdown of the criteria for limiting the sample frame and the number of participants meeting each successive criterion.

Chart 5. Criteria for the sample frame and number of participants meeting criteria

Criteria for narrowing sample frame	Comprehensive and Targeted faculty		Institute faculty
	Number excluded by criteria	Number remaining	Number excluded by criteria	Number remaining
Total number of faculty participating in MSP	n/a	1,428	n/a	117
STEM disciplinary faculty	565	863	44	73
Tenured or tenure-track faculty	208	655	21	52
Faculty spending 80 or more hours on MSP in each year	389	266	8	44
Faculty who can be categorized as either a Stayer or Leaver	39	227	3	41
Faculty at Doctoral-granting Universities*	118	109	n/a	n/a
Total	1,319	109	76	41

*This criterion will only be applied to Comprehensive and Targeted faculty.

B.2. Information Collection Procedures/Limitations of the Study

Because the MSP MIS does not contain names or contact information, the Principal Investigator (PI) of each MSP project (who is required to maintain such contact information) will be asked to contact prospective respondents to ascertain if they are willing to participate in a customer satisfaction interview. If a selected participant agrees, the PI will provide Westat researchers with the participant’s contact information.

Participants will first be contacted by email to schedule a telephone interview. The date and time of the interview will be convenient to the participant. At the scheduled interview time, a member of the Westat research team will call the participant by telephone. The interviews will follow the attached telephone interview protocol. This method offers systematic collection of data across participants, while allowing the most accurate interpretation of participants’ responses. The interviewer will maintain notes of the interview.

B.2.1. Statistical Methodology for Stratification and Sample Selection

Within the refined sample frame of 109 Comprehensive/Targeted and 41 Institute faculty, a stratified random sampling procedure will be used. In addition, we will randomly select 4 of the 13 new STEM faculty and 4 of the 14 veteran STEM faculty in Comprehensive/Targeted projects who attained tenure since beginning their participation in MSP. The strata are shown in Chart 6, with the number of participants in each stratum to be randomly selected. In cases where a respondent elects not to participate in the telephone survey, we will randomly select someone else from within the appropriate strata.

Chart 6. Sampling strata with corresponding cell size

Program	Characteristics	Prior experience in K-12 education projects		Total
		New	Veteran
Comprehensive and Targeted	Stayers	10	10	20
	Leavers	10	10	20
	Tenure track to tenure	4	4	8
Institute	Stayers	4	4	8
	Leavers	4	4	8
Total		32	32	64

B.2.2. Estimation Procedure

Not Applicable

B.2.3. Degree of Accuracy Needed for the Purpose Described in the Justification

Not Applicable

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures

Not Applicable

B.2.5. Use of Periodic (Less Frequent Than Annual) Data Collection Cycles

Not Applicable

B.3. Methods for Maximizing the Response Rate and Addressing Issues of Nonresponse

Once Westat has drawn a preliminary sample, the Principal Investigator for the corresponding MSP project will be asked to contact the prospective respondents to ascertain if he or she is willing to participate in a customer satisfaction interview. If a selected participant agrees, the PI will provide Westat researchers with the participant’s contact information. In cases where a respondent elects not to participate in the telephone survey, Westat will randomly select someone else from within the appropriate strata.

 

B.4. Tests of Procedures or Methods

Not applicable.

B.5. Names and Telephone Numbers of Individuals Consulted

Agency Unit

Philis Hauser, NSF, 703-292-5104

 

Contractor

Joy Frechtling, Westat, 301-517-4006

ATTACHMENTS TO SECTION B: Draft Interview Protocols for Stayer STEM IHE Faculty, Leaver STEM IHE Faculty, Tenure Achieving STEM IHE Faculty