Appendix D - Site Visit Procedures, etc.

APPENDIX D
Description of MSP-PE’s Site Visit Procedures, Site Visit Training, and
Structure of the Site Visit Report
I. MSP-PE’s Site Visit Procedures, p. D-1
II. MSP-PE’s Site Visit Training, p. D-3
III. Structure of the Site Visit Report, p. D-5

I. MSP-PE’s SITE VISIT PROCEDURES

I A. Illustrative Site Visit Routine (3 visitors for 3 days)
Average Burden
(in Hours) for each
Type of Interviewee*
PI
PC
P

PE

Day 1
early a.m. or night before: travel
ALL THREE VISITORS:
12:30-3:30p.m. (with working lunch):
-principal investigator (PI) and project coordinator (PC)
(2 interviewees X 3 hrs. each)
3:30-5:30p.m.
-partnership evaluator (PE)
(1 interviewee X 2 hrs.)
Day 2
SITE VISITORS 1 AND 3:
8:00-10:00a.m.
-team reviews and copies retrieved documents and
qualitative and quantitative data from Day 1
10:00a.m.-1:00p.m.
-principal investigator (PI), project coordinator (PC), and
partnership evaluator (PE) jointly clarify partnership’s
experiences and data
(3 interviewees X 3 hrs. each)
2:00-5:00p.m.
-partnership evaluator (PE) review of selected evidence-based
data collection activities
(1 interviewee X 3 hrs.)
SITE VISITOR 2 (can rotate with Site Visitor 1):
8:00a.m.-12:00
-separate interviews of first set of four partners (P)
(4 interviewees X 1 hr. each)
1:00-5:00p.m.
-separate interviews of second set of four partners (P)
(4 interviewees X 1 hr. each)
Day 3
8:00-11:00a.m.
-team reviews and copies retrieved documents and
qualitative and quantitative data from Day 1,
also preparing for final interviews
11:00a.m.-12:00
-principal investigator (PI) and project coordinator (PC)
(2 interviewees X 1 hr. each)
p.m.: travel

3

2

3

D-1

3

3

3

4

4

1

* PI = principal investigator; PC = project coordinator; P = partner; and PE = partnership evaluator

COSMOS, November 8, 2010

3

1

I B. Cross-Walk Between People to be Interviewed and Topics
in the Site Visit Instruments

Questions

(1)
MSP Principal
Investigator

(2)
MSP Project
Coordinator

(3)
MSP
Evaluator

(4)
MSP Partner
(n=8)

A1
A2
A3
A4
A5

Day 1
Day 1
Day 1
Day 1
Day 1

Day 1
Day 1
Day 1
Day 1
Day 1

Day 1
Day 1
Day 1
Day 1
Day 1

Day 2
Day 2
SKIP
Day 2
Day 2

B1
B2
B3

Day 1
SKIP
Day 1

Day 1
SKIP
Day 1

Day 2b
Day 2b
Day 2b

SKIP
SKIP
SKIP

C1
C2

Day 2a
Day 2a

Day 2a
Day 2a

Day 2a
Day 2a

SKIP
SKIP

D1
D2

Day 2a
Day 2a

Day 2a
Day 2a

Day 2a
Day 2a

SKIP
SKIP

E1
E2

Day 2a
Day 2a

Day 2a
Day 2a

Day 2a
Day 2a

SKIP
SKIP

F1
F2

Day 3
Day 3

Day 3
Day 3

Day 2b
Day 2b

Day 2
Day 2

Day 2a: Interviews held together
Day 2b: Project evaluator interview held separately

COSMOS, November 8, 2010

D-2

II. MSP-PE’s SITE VISIT TRAINING

II A. INTRODUCTION TO THE SITE VISIT INSTRUMENTS
The MSP-PE site visit instruments contain a set of questions directed to the site visit
team, not any given interviewee. The team is to keep these questions—or line of inquiry—
in mind, while collecting converging evidence from multiple sources (e.g., discussions with
key MSP members, review of MSP documents and data, and direct field observations),
accessed during a site visit. The team’s answers are then to form the basis for its site visit
report.
Related Information about the MSPs. The MSP-PE’s site visits are but one source of
data available about the MSPs. Other major sources include: 1) the MSP-MIS; 2) MSP
grant reports; and 3) MSP-produced evaluation or research reports. Site visit teams will
review these other sources thoroughly, prior to conducting a site visit.
At the same time, the information in these other sources needs to be recognized as
“self-reported.” In this sense, any reported accomplishments may in fact be considered
“claims.” To this extent, the site visit instruments require the site visit team to ask
corroboratory questions, including efforts to collect copies of the MSPs’ own instruments
and data on various key topics.
Specializations of Site Visit Members. Team members assembled for any given site
visit will cover as many specialties as possible (e.g., curriculum, instruction, or professional
development; evaluation; equity of educational outcomes; partnerships; or science and
engineering careers). In addition, all team members will participate in MSP-PE specific
training and orientation.

II B. SITE VISIT PREPARATION
Prior to making site visits, all team members will consult and assimilate the following
materials:
1. For collecting and documenting information from multiple sources of evidence,
see Yin, Case Study Research: Design and Methods, 2009, Chapter 4.
2. For an overall orientation to field-based evaluations, see U.S. General Accounting
Office, Case Study Evaluations, Washington, DC, Nov. 1990, Transfer Paper
10.1.9.
3. For information on the specific MSP site to be visited, read the latest annual
grant and evaluation reports and other readily available papers by the MSP
(e.g., papers prepared for NSF’s MSP Learning Network conferences).
COSMOS, November 8, 2010

D-3

Training. Prior to the onset of any of the site visits, members will participate in an
initial training session, covering the site visit protocol and general evaluation methods.
Orientation. One week prior to every site visit, team members will hold a meeting
(e.g., a conference call) to discuss the objectives for the forthcoming site visit and identify
the topics likely to be covered by the site visit. The orientation should be preceded by
members having reviewed the existing documents about the specific MSP (see #3 above).

II C. DIVISION OF WORK AMONG SITE VISIT MEMBERS
At the outset of each site visit, each site visit team will assign its individual members
to the following specific roles, functions, and responsibilities.
Each site visit member will be formally responsible for covering specific portions of
the site visit instruments. The responsibility includes making sure that the relevant data
have been addressed and collected, and later composing that portion of the site visit team’s
report. In a rotating manner, the other members of the site visit team will support the
“lead” member in each of these portions, by: 1) carrying out the needed fieldwork; 2)
providing notes and other materials to the lead member to support the analysis and writing;
and 3) reviewing the draft portion and providing feedback on the draft.
All members of the team will contribute to the draft report. The team also will assign
one of its members to coordinate and lead the report drafting. This person’s main objective
will be to assure the integrity of the report as a whole, eliminating redundancies and
resolving disagreements that might arise because different portions will have been written
by different team members.

COSMOS, November 8, 2010

D-4

III. STRUCTURE OF THE SITE VISIT REPORT

The site visit report will be a narrative (with tables and exhibits, where relevant)
organized according to the six main sections of the site visit instruments: A) partnerships,
B) evidence-based design and outcomes, C) teacher quality, quantity, and diversity, D)
challenging courses and curricula, E) role of the IHE disciplinary faculty, and F) rival
explanations. Within each section, the narrative should follow the sequence of numbered
interview questions (A1, A2, A3, etc.), bringing together the responses from the different
interviewees under the same question number and hence also discussing the consistencies
and inconsistencies across the interviewees.
At the end of the report should be listed alphabetically, in separate sections, the full
citations of any documents used and the persons (name, title, place, and date) interviewed
to complete the site visit report. Footnotes in the narrative should cite the items in these
two lists by coded number, to show the source(s) of the information in the narrative. The
coded numbers should be assigned to the sources randomly, cutting across both documents
and interviewees, but the decoding of the numbers should not be part of the report. Rather,
the decoding information will be kept in secure project files, to be available only to the
author of the site visit report and the project director of the national evaluation. This
coding procedure therefore retains the anonymity and confidentiality of the original
interviewees.
If an MSP’s tables and exhibits are part of the report, these materials should not be recomposed or re-formatted in any way. The materials should be presented in their original
form (e.g., as a PDF copy), with citations added to identify their source. The creation of
new exhibits, figures, and tables by the site visit team itself is welcome but should not be
considered a requirement and should be resisted if leading to any delay in completing the
narrative report.
The team should write the narrative in plain English: limited jargon, active (not
passive) voice, third (not first or second) person, and simple (not compound) sentences.
For instance, simple sentences do not usually contain relative pronouns. The writing also
should follow a glossary of MSP-specific terms and acronyms, to be shared ahead of time
with the evaluation team.

COSMOS, November 8, 2010

D-5