CROSSWALK of GK-12 Evaluation Topics to Indicators and Data Sources

March 24, 2008

CROSSWALK of GK-12 Evaluation Topics to Indicators and Data Sources

This document outlines the information we will collect in order to address the GK-12 Program Evaluation Research Questions (see Appendix A at the end of this document for a list of the study research questions). For each primary study area (Fellows; K-12 Teachers and Students; and participating IHEs) we have outlined the primary outcomes of interest and relevant covariates – that is, other factors we hypothesize may influence the outcomes of interest.

For example, one outcome of interest to this study are the professional teaching skills gained by Fellows as a result of participating in GK-12. Covariates that we will examine and control for when analyzing Fellows’ teaching skills include Fellows’ previous teaching experiences and non GK-12 teaching experiences during graduate school. The indicators described in this document are the topics that will be addressed; a given topic may correspond to a single survey question or multiple questions.

As you will see, the core sources of information for this study are as follows:

• To examine impacts on FELLOWS, the primary sources of information will be the surveys of FELLOWS/COMPARISON INDIVIDUALS and surveys of FACULTY ADVISORS.

• To examine program outcomes for TEACHERS, the primary source of information will be the survey of TEACHERS and individual project EVALUATION REPORTS.

• To examine program outcomes for graduate education at participating IHEs, the primary sources of information will be the survey of PIS, survey of FACULTY ADVISORS and the site visit/telephone INTERVIEWS.

Terminology: Note that due to the nature of the comparison group for Fellows, we can assess GK-12 program impacts for Fellows, but in other areas we will examine program outcomes as reported by participants but not causally attributed to GK-12 through a quasi-experimental comparison group.

KEY:

■ Primary data source

□ Secondary data source

Fellows * = Descriptive qualitative data

TOPIC	FELLOW INDICATORS	F-En	F-Gr	C	PI	T	FA	Int	SV	DM	ER
COVARIATES	 report descriptively & control for during outcomes analysis
Fellows’ previous experiences and	Career goals before GK-12 participation (interest in a STEM career, motivation for pursuing STEM graduate degree)	■	■	■
characteristics	Interest in STEM Education	■	□	■
	Demographics (gender, ethnicity)	■	■	■						■
	Reasons for applying for GK-12 (including reasons for choosing GK-12 over other potential funding sources such as RA-ships)	■	■	■
	STEM discipline of study	■	■	■
	Prior teaching experiences (K12 schools, higher education, informal education, adult education, volunteering, tutoring, mentoring)	■	□	■
Other graduate school experiences	Amount and type of non-GK-12 funding received before and after GK-12 participation	■	■	■
	Other teaching experiences (TA, K-12 involvement, when occurred relative to GK-12 participation)	■	■	■
	Non-GK-12 training received in graduate school in teamwork, communication, teaching, research	■	■	■
	Ever applied for GK-12 Fellowship (why or why not)			■
Fellows’ GK-12 experiences	Nature of training received through GK-12: Topics: teamwork, communication, teaching, research Format: workshop, seminar, coaching, mentoring Timing: summer prior, during school year Duration and intensity	■	■		□
	GK-12 activities in classroom or other STEM educational settings
	Year in program when participated (1st year, 2nd year, etc.)	□	□	□						■
	Number of years as a Fellow (1, 2, etc.)	□	□	□						■
Fellows’ GK-12 experiences (cont.)	Characteristics of school(s) in which Fellows work (Elem/Mid/High; SES; etc.)									■
	Number of teachers with whom Fellows work	■	■							■
	Characteristics of teachers with whom Fellows work (addressed in teacher covariate section)					■				■
	Time devoted to and nature of GK-12 activities (teaching, unit or lesson module development, reviewing student work, etc.)	■			□	□
	Level of autonomy and responsibility during GK-12 activities	■				□
	Innovation versus accommodation (degree to which F. brings own research into classroom vs. work with existing topics and curriculum)	■			□	□
	Nature of disciplinary GK-12 experiences – how Fellow connects own research to education and/or uses own expertise during GK-12 activities	■				□
	Nature of collaboration with teacher	■				□
	Fellow interactions with other Fellows and other graduate students	■		■	□
	Sense of GK-12 community: extent to which Fellows interact with and support each other	■	■		□
Support of academic advisor	Advisor participation in GK-12 (Yes/No; if Yes, nature of participation) (e.g., visit the school with the Fellow)	□	□				■
	Advisor support for Fellow’s involvement in GK-12 and GK-12-like activities	■	■	■			■
Project recruitment of GK-12 Fellows	Desired characteristics, selection criteria (e.g. teaching experience)				■					□1
Larger institutional context of GK-12 award	Sense of academic community – extent to which graduate students in department feel supported by each other	■	■	■	■		□
	Support and encouragement from department faculty and graduate students for GK-12 activities	■	■	■	■		■
	Engagement in GK-12 award by faculty members in participating departments and beyond	■	■		■		□
	Institutional characteristics (type, size, number of graduate students, presence of a School of Education)
FELLOW OUTCOMES	 report descriptively & compare with Comparison groups of enrolled and graduated non-GK-12 students, controlling for influencing factors. Important to analyze by STEM discipline.
Professional preparation in teamwork, communication,	Demonstrated use of professional skills (teamwork, communication, teaching, and research) - post Fellowship, pre graduation - post graduation	■	■	■		□	□	□			*
teaching, and research	Perceived benefits of graduate experiences (including GK-12 and other)	■	■	■			■	□			*
Academic progression	Length of time to complete degrees	■	■	■				□
and retention towards degrees	Impact of graduate experiences (including GK-12) on increasing or decreasing time to degree	■	■	■			□	□
	Proportion of students (Fellows, Comparison) that graduate from their degree programs		■	■				□		■	■
	Impact of graduate experiences (including GK-12) on graduating with degree	■	■	■			□	□
	Other impacts GK-12 has on Fellows’ research progress and productivity	■	■	■			■	□
Career choices and	Current career goals	■	■	■				□
advancement	Impact of graduate experiences (including GK-12) on career goals	■	■	■				□
	Proportion of graduates (Fellows, Comparison) that go on to pursue a PhD or postdoctoral position		■	■				□
	Extent to which graduates pursue STEM careers		■	■				□
	Choice of careers		■	■				□
Career choices and	Time it takes to get a job after leaving the GK-12 institution		■	■				□
advancement (cont.)	Competitiveness in getting a job		■	■			□	□
	Role of graduate experiences (including GK-12) in obtaining a job		■	■				□
	Percent who are employed		■	■				□
	Types of positions held		■	■				□
	Careers pursued		■	■				□
	Responsibilities and leadership capabilities in careers		■	■				□
Understanding of the	Extent of knowledge of STEM educational issues	■	■	■		□	□	□
responsibilities of scientists	Engagement with science education-related outreach and policy	■	■	■		□	□	□

K-12 Teachers, Students, and Schools

TOPIC	K-12 TEACHER AND STUDENT INDICATORS	F-En	F-Gr	C	PI	T	FA	Int	SV	DM	ER
COVARIATES	 report descriptively & control for during outcomes analysis
Teachers’ previous	Type of certification/subjects certified to teach					■
experiences and	Number of undergraduate and graduate level STEM courses taken					■
characteristics	STEM classes/content taught before GK-12					■
	Advanced degree					□				■
	Number of years of teaching experience					□				■
	Previous STEM work and/or research experiences					■
	Demographics (gender, ethnicity)					■
	Career goals					■
	Reasons for participating in GK-12					■
Teachers’ GK-12	Number of years involved with GK-12									■
experiences	Number of GK-12 Fellows have worked with									■
	Nature of interactions with other GK-12 Fellows, Teachers, and IHE faculty members				□	■
	Nature of training received through GK-12: Topics: teamwork, communication, teaching, research Format: workshop, seminar, coaching, mentoring Timing: summer prior, during school year Duration				□	■
	Nature of GK-12 activities (interactions with Fellows, attending GK-12 meetings, etc.)				□	■
Teachers’ previous or concurrent STEM reform, professional development experience	Other STEM reform and professional development experiences (e.g., MSP, Eisenhower, specific materials or kits e.g. Foss)					■
School context	Percent of school colleagues who have also participated in GK-12					■
	GK-12 saturation model: Whole school, whole grade, or selected teachers within school				■
	Extent to which school is engaging in STEM education reform				■
K-12 OUTCOMES	 report descriptively and triangulate among multiple data sources
Teachers’ STEM content knowledge	Perceptions of changes in knowledge of STEM content (related to current teaching assignment and/or beyond)	□	□			■		□			*
	Perceptions of changes in knowledge of STEM research	□	□			■		□			*
	STEM instruction (depth and breadth of topics taught)					■		□			*
	Increased confidence in teaching STEM content					■		□			*
	Perceptions of how GK-12 influenced interest in pursuing additional STEM content knowledge (i.e., taking continuing education credits, taking graduate courses in STEM, seeking Board certification in STEM)					■		□			*
Teachers’ use of STEM pedagogical	Perceptions of changes in knowledge of pedagogical practices and tools	□	□			■		□			*
practices & tools	Confidence in using STEM pedagogical practices and tools (e.g. technology such as software, lab equipment, etc.)	□	□			■		□			*
	Use of STEM pedagogical practices					■		□			*
	Use of STEM pedagogical tools (e.g. technology such as software, lab equipment, etc.)	□	□			■		□			*
Teachers’ participation in	Involvement in PD opportunities related to STEM content and pedagogy (e.g. presentations at conferences; writing papers; conducting workshops)					■		□			*
STEM education activities	School leadership activities (e.g., STEM department chair, STEM lead or resource teacher, district STEM coordinator, participation in STEM committees, leading science/math clubs, mentor programs, school or committees or task forces)					■		□			*
	Involvement in community activities such as community enhancement projects (e.g. museum volunteer, recycling project) STEM policy activities					■		□			*
	Sense of membership in a STEM education community (Involvement in professional organizations, mentoring new teachers)					■		□			*
Teachers’ access to STEM educational curricula and resources	Access to STEM educational curricula and resources (e.g., new classroom curricula – labs, units, modules; material resources—lab equipment, computers, and other supplies, internet, bibliographies, catalogs, human resources)				□	■		□			*
Students’ gains (knowledge,	Changes in student knowledge and understanding of STEM content and research	□	□			■		□			■
interest)	Changes in student interest in STEM extracurricular activities such as science clubs and community based organizations such as environmental groups	□	□			■		□			■
	Changes in student interest in taking advanced STEM courses (middle and high school) and pursuing careers in STEM	□	□			■		□			■
	Perception of Fellows as role models	□	□			■		□			■
	Student interest and engagement during Fellow’s participation in classroom	□	□			■		□			■
	Perceptions of changes in student engagement with informal science activities (e.g., watch science-focused TV shows, go to science museums, read science books or magazines, university lectures, meetings)	□	□			■		□			■

Institutions of Higher Education

TOPIC	IHE INDICATORS	F-En	F-Gr	C	PI	T	FA	Int	SV	DM	ER
COVARIATES	 report descriptively & control for during outcomes analysis
IHE proximity to partner schools	How much time Fellows spend traveling to schools Facility with which teachers and K-12 students can make it to campus if needed	□			■	■
Previous Partnerships with K-12 Schools and teachers	Other outreach and partnership programs with K-12 schools within the department				■			□	■
Length of time IHE has been involved with GK-12 Program	Number of years GK-12 program has been run; renewal of the program Track 1/Track 2 versus 5 year									■
Value placed on education and outreach in the department	Tenure trifecta breakdown (research, education, service), teaching/outreach awards or any other departmental recognition				■		■	□	■
Level of interdisciplinary collaboration if GK-12 program spans different departments	Cross-departmental/interdisciplinary interactions among GK-12 Fellows and faculty members	□	□	□	■		■	□	■
Other programs in department/school with similar goals	Presence of other NSF programs focusing on education, other outreach programs already run through department/school/institution Gk-12 partnership with other initiatives (MSP, etc.)				■			□	■
IHE GRADUATE EDUCATION OUTCOMES	 report descriptively and triangulate among multiple data sources
Incorporation of GK-	Availability of GK-12 like activities to other students			■	■		■	□	■		*
12-like activities in STEM degree	STEM faculty incorporate GK-12 like activities in their training of graduate students	■	■	■	■		■	□	■		*
programs at IHEs	GK-12 activities continue after funding (Sustainability)				■		■	□	■		*
Institutional commitment to the program	Financial commitment of university to GK-12 project, including funding for additional Fellows before or after funding period ends				□			□	■		*
Partnership between IHEs and K-12	Number of faculty and staff who interacted or partnered with schools prior to GK-12, and who do so now				□			□	■		*
Schools	Frequency with which faculty or staff help teachers develop activities, solve a problem, or provide materials/supplies to teachers; volunteer in the schools; or involve teachers in their research				□			□	■		*
Extent to which GK-12 award develops,	Extent to which faculty, departments, or university sponsor or participate in joint activities with schools/district				□			□	■		*
strengthens, and sustains partnerships	Extent to which other STEM graduate students have become engaged with the schools since start of GK-12 project				□			□	■		*
between IHEs and K-12 schools	Extent to which other IHE STEM departments become engaged with K-12 schools as a result of the relationships developed through GK-12 (e.g. outreach programs, service learning)				□			□	■		*
	Teachers feel part of a larger STEM education community							□	■		*
	STEM departments maintain a relationships with schools and teachers after NSF funding ends (completed projects only)				□			□	■		*

Appendix A: Research Questions

Fellow Outcomes:

1. What is the impact of participating in GK-12, both while enrolled and after graduation, for Fellows’

   1. Professional preparation? (teamwork, communication, teaching, collaboration, and research)

   2. Academic progression and retention towards their degrees?

   3. Career choices and advancement?

   4. Understanding of the responsibilities of STEM professionals for outreach and social awareness?

K-12 Outcomes (Teachers, Students, Schools):

2. What are the outcomes of participating in GK-12 for K-12 teachers’

   1. STEM content knowledge?

   2. Use of STEM and GK-12 related pedagogical practices and tools?

   3. Participation in STEM professional development and subsequent GK-12 related activities?

   4. Access to STEM educational curricula and resources? (e.g., classroom resources, collaborative opportunities etc.)

3. What are the outcomes of participating in GK-12 for K-12 students’ knowledge of and interest in STEM fields and STEM-related careers?

Outcomes for Graduate Education at Participating IHEs:

4. To what extent have GK-12 inspired ideas and practices been incorporated into the professional preparation of STEM graduate students at participating institutions of higher education?

5. To what extent have participating STEM faculty members and university staff developed, strengthened, and sustained partnerships with local school districts?

Analysis across all of these research questions will focus on two dimensions – 1^st, establishing the influence of GK-12 in effecting change, and 2^nd, determining the characteristics of GK-12 awards which are associated with higher or lower levels of influence. In other words, what makes for an effective / successful GK-12 project? What makes for a beneficial experience for Fellows and Teachers? Etc.

1 New question as of 2007.

1