3145-0200 Part B

B. COLLECTION OF INFORMATION
EMPLOYING STATISTICAL METHODS
B.1. Respondent Universe and Sampling Methods
The entire universe of the MSP Program’s partnerships is included in the data
collection (48 in Cohorts 1-3 were covered in the original OMB clearance, and 32 in
Cohorts 4-6 are to be covered under this new request). Statistical sampling will not be
used, as all of the 32 new partnerships will be site visited over a three-year period. Within
each of 32 partnerships are 11 relevant staff and officials (previously enumerated in
A.12.1), for a total of 352 respondents.
Each respondent will provide answers to the instruments only once during the threeyear life of this OMB clearance period. We expect response rates of 95 percent for
interviews (a 95 percent rate had been achieved under the earlier clearance, covering the
original 48 partnerships). The data will be collected through a series of site visits as well
as accessing existing archival sources (e.g., partnerships’ annual reports and evaluation
reports). Appendix D contains the procedures to be used in conducting these site visits,
indicating how they are aimed at collecting data from multiple sources of evidence.
B.2. Procedures for the Collection of Information
For each of 32 partnerships, data will be collected from the lead partnership staff
(principal investigator and project coordinator), other partnering staff, and partnership
evaluator. The site visit procedures in Appendix D provide an illustrative site visit routine
in which three persons will be conducting a three-day site visit. The limitation of this
evaluation is that it is a correlational design and as such will not yield causal conclusions.
COSMOS, November 8, 2010

31

B.2.1. Statistical Methodology for Stratification and Sample Selection
No statistical methodology for stratifying and selecting a sample has been used,
because the planned data collection will cover the entire census of Cohort 4-6 partnerships.
However, statistical methods have and will again be used to analyze the findings from the
data collection. These methods include multivariate models (e.g., regression analyses) and
their accompanying tests of statistical significance.
B.2.2. Estimation Procedure
No estimation procedure is required because the planned data collection will cover the
entire census of Cohort 4-6 partnerships.
B.2.3. Degree of Accuracy Needed for the Purpose Described in the Justification
The current request for OMB clearance involves data collection that will cover the
entire census of Cohort 4-6 partnerships in the program being evaluated—the MSP
Program. As such, the census provides a high degree of accuracy regarding the conditions
of the Program—implementation, outcomes, and success and failure.
B.2.4. Unusual Problems Requiring Specialized Sampling Procedures
The nature of the structure of the partnerships demands a special sampling procedure
for partner interview selection. A required partnership in the MSP Program is between one
or more institutions of higher education (IHE) or eligible nonprofit organizations (or
consortium of such institutions or organizations) and one or more K-12 systems—i.e., local
education agencies (LEAs). Beyond these two core partners, the partnership also may
include other partners—e.g., a state educational agency (SEA), one or more businesses, and

COSMOS, November 8, 2010

32

community organizations. Thus, the MSP Program distinguishes between core and noncore partners.
Only the core partner or partners share the responsibility and accountability for the
partnership award. All core partners also are required to provide evidence of their
commitment to undergo the coordinated institutional change necessary to sustain the
partnership effort beyond the funding period. A non-core or supporting partner is not
required to commit to the institutional change necessary to sustain award activities beyond
the funding period but is an important stakeholder in K-12 math and science education.
Given this distinction, the data collection priority will be given to the core partners. The
interview sample will accommodate a total of nine partners: the lead IHE or K-12
institution (principal investigator and project coordinator), and eight other partners (coprincipal investigators and partners), with the core partners receiving the priority among
these eight other partners.
B.2.5. Use of Periodic (Less Frequent Than Annual) Data Collection Cycles
In this evaluation it is not necessary to collect data from respondents more than once.
Each of the data collection activities asks respondents to provide answers to the site visit
instruments just once.

COSMOS, November 8, 2010

33

B.3. Methods to Maximize Response Rates and to Deal with Issues of Non-Response
All of the respondents are direct participants in a partnership award, and the
conditions of these awards include cooperating with this program evaluation’s data
collection efforts. In addition, the planned site visits will be arranged to suit the
convenience of the respondents and their schedules. Thus, barring last minute illnesses or
other emergencies that may occur just before a scheduled site visit takes place, all
respondents will be available, and a high response rate is anticipated.
Should some interviewees be absent or unable to attend a site visit, the response rate
will be maximized by using follow-up telephone calls. During these follow-up calls, to
take place after a site visit has been completed, the original site visit team will administer
the site visit instruments, but telephonically.
Together, the data collection during the original site visits and in any follow-up
telephone calls should produce a 95 percent response rate from all respondents.
B.4. Tests of Procedures or Methods
The field procedures and the earlier versions of the site visit instruments were
successfully used under the earlier OMB clearance, to collect data from the original 48
partnerships in Cohorts 1-3. All site visits took place as planned (some of them had to be
re-scheduled but were eventually held), and in a few cases respondents who were
unavailable due to last minute emergencies were interviewed telephonically after the site
visit. The collected data have been analyzed and reported, as described in the introduction

COSMOS, November 8, 2010

34

to this supporting statement. In this sense, the planned procedures and methods have been
successfully tested.
B.5. Names and Telephone Numbers of Individuals Consulted
The aforementioned group of external experts (see earlier response to Q. A8) have
been consulted throughout the evaluation regarding its statistical procedures and design.
These experts have met three times as a plenary group and also have provided numerous
written reviews of the evaluation’s reports and work. Their names and telephone numbers
are as follows:
Robert Boruch, Ph.D.

(215) 898-0409

Sharon Johnson Lewis

(248) 569-4802

Douglas Osheroff, Ph.D.

(650) 723-4228

Charles S. Reichardt, Ph.D.

(303) 871-3783

Warren Simmons, Ph.D.

(401) 863-7675

Mary Lee Smith, Ph.D.

(480) 965-6357

Philip Uri Treisman, Ph.D.

(512) 471-6190

Alan Tucker, Ph.D.

(631) 632-8365

The person who will collect and analyze the data and who will supervise all other
persons involved in the data collection and analysis is the project director of the program
evaluation: Robert K. Yin, COSMOS Corporation, 240-223-5200.

COSMOS, November 8, 2010

35