Evaluation Construct Matrix

Appendix V
Evaluation Construct Matrix

Study Questions to be Addressed in the Evaluation of the ECEPD Program
Data Sources

Topic

Proj. Goals

Study Questions

What are the major goals of the ECEPD projects?

In-Depth
Project and
interviews
Evaluation
Grantee
Grantee
In-depth
with
Description
Grantee
Performance Evaluation interviews with Evaluation
Forms
Applications
Reports
Reports Project Teams
Teams

Addressed
to all
grantees or
subset?

Project
Directors'
Interview

All

1

x

Review of
Project
Materials

Proj.
Structure

What partners are included in ECEPD projects, and
what are their roles?

All

2,3

x

Proj.
Structure

What are the infrastructures for delivering
professional development that have been extended
or created by the ECEPD projects?

Subset

21

x

Proj.
Structure

How are ECEPD projects monitoring and evaluating
the impact of their projects?

All

29

x

Proj.
Structure

What strategies are ECEPD grantees using to
sustain their projects after the end of the grant?

All

36,37

All

2

7

All

4,5

2,4

Subset

6

Subset

7

Proj.
educators? Which strategies are most successful?
Implement.

All

10,11,12

x

x

How do ECEPD projects recruit and retain staff to
Project
Implement. deliver professional development?

All

8,9

x

x

Subset

33,34

x

x

What state and/or local policies that need to be
taken into account to understand implementation
Proj.
Implement. and outcomes for ECEPD grantees?

Proj.
Structure

How are relationships across ECEPD partnerships
structured? What structural factors contribute to
successful partnerships?

Proj.
Structure

Which evaluation designs are most feasible and
informative?

x

5,10,
12,13

x

How do ECEPD projects adapt over time? What
Proj.
common issues arise that require adaptations or
Implement. modifications?

11,12

What strategies do ECEPD projects use to recruit

What factors are linked to successful
implementation of ECEPD projects?
Proj.
Implement.

Child Trends - May 9, 2007

3, 11,16

1

Study Questions to be Addressed in the Evaluation of the ECEPD Program

Topic

Study Questions

Proj.
Activities

What sets of professional development activities are
offered by ECEPD grantees? Can the various sets
of activities be grouped to identify distinct types of
professional development strategies?

Proj.
Activities

What supports are ECEPD grantees including in
their projects to help educators enroll and progress
in their programs?

Proj.
Activities

How are the professional development activities
chosen or developed? How do projects access the
research in making their choices?

Proj.
Activities

Proj.
Activities

What is the content of professional development
activities being implemented by ECEPD grantees?
How is content monitored?

Who provides professional development in ECEPD
projects? What is their level of training and
education? To what extent is their teaching/training
monitored by the ECEPD project?

In-Depth
Project and
interviews
Evaluation
Grantee
Grantee
In-depth
with
Description
Grantee
Performance Evaluation interviews with Evaluation
Forms
Applications
Reports
Reports Project Teams
Teams

Addressed
to all
grantees or
subset?

Project
Directors'
Interview

All

14-18

x

x

Subset

13,25,26

x

x

Subset

21

x

Subset

20

x

8

6,7

x

Subset

8

x

9,10

6,7

x

All

10,22,24

x

x

x

x

All

10

x

x

x

x

All

30

x

Subset

13,25,26

x

Review of
Project
Materials

13,15
6,7

x

What is the age, education, and experience level of
Charac. of participants in ECEPD projects?
Participants

In what types of early education and care settings
Charac. of do the educators work? What are the
Participants characteristics of the children they serve?

x

How does the program track/document individual

Charac. of participants' progress through the program?
Participants

Charac. of What are the major challenges educators face?
Participants/ How do ECEPD projects address and support
Supports educators with particular challenges?
and
Incentives

Child Trends - May 9, 2007

13,15

2

Study Questions to be Addressed in the Evaluation of the ECEPD Program

Topic

Study Questions

Do ECEPD projects change educators’
understanding of the terminology and concepts in
Educator
specific content areas and of children’s
Outcomes
development within these content areas?
Do ECEPD projects change educators’ practices
within the classroom or group, including
Educator implementation of curriculum activities, making
Outcomes changes in the physical environment and routines,
and changes in the quality of interactions?
Do ECEPD projects change educators’ enjoyment of
and sense of competence in teaching specific
Educator content areas as well as understanding of the
Outcomes importance of children’s learning in a particular
area?
Do ECEPD projects change educator’s professional
development status, including changes in the hours
or credits of education attained, degrees and
Educator certification achieved, wages and/or benefits
Outcomes received, and professional recognition in terms of
levels on a career lattice, a quality rating system, or
system of tiered reimbursement?
Educator
Outcomes

Are changes in educators’ knowledge and/or
practice sustained over time?

Educator
Outcomes

Which educators are most likely to improve their
knowledge, skills and/or professional development
status?

Do ECEPD projects improve children’s outcomes?
Are certain child outcomes more likely than others
Child
to be affected by educator’ professional
Outcomes
development?

Child
Outcomes

Which children are most likely to experience
improvements in their outcomes?

Addressed
to all
grantees or
subset?

Project
Directors'
Interview

In-Depth
Project and
interviews
Evaluation
Grantee
Grantee
In-depth
with
Description
Grantee
Performance Evaluation interviews with Evaluation
Forms
Applications
Reports
Reports Project Teams
Teams

Subset

x

x

x

17

8,9

Subset

x

x

x

17

8

Subset

x

x

x

8

x

x

x

8

Subset

x

x

x

8

Subset

x

x

x

14

8

Subset

x

x

x

17

9

Subset

x

x

x

Subset

19

Child Trends - May 9, 2007

Review of
Project
Materials

9

3

Study Questions to be Addressed in the Evaluation of the ECEPD Program

Topic

Study Questions

Addressed
to all
grantees or
subset?

Project
Directors'
Interview

In-Depth
Project and
interviews
Evaluation
Grantee
Grantee
In-depth
with
Description
Grantee
Performance Evaluation interviews with Evaluation
Forms
Applications
Reports
Reports Project Teams
Teams

Review of
Project
Materials

What are the emerging patterns in the existing
literature regarding the professional development
Promising practices that have positive outcomes for educators'
(lit review)
Practices knowledge and skills and child outcomes?

How can the evidence from ECEPD grantee
evaluations be integrated with the existing literature
in professional development for early childhood
Promising
educators to extend our knowledge? What
Practices
promising directions are emerging?

Are certain professional development strategies
more likely than others to be linked to improved
Promising
educator and/or child outcomes?
Practices

Subset

All

39

Child Trends - May 9, 2007

x

18

15

x

17

14,15

x

4