Provider Administrator--Other

Provider/Administrator—Other

Attachment B: System of Care Assessment

Overview of the System of Care Assessment Framework

The Comprehensive Community Mental Health Services for Children and Their Families
Program, funded by the Center for Mental Health Services (CMHS), provides grants to states,
communities, and American Indian Tribes to improve and expand their service delivery systems
to meet the needs of children and families. This services initiative is built on the Child and
Adolescent Service System Program (CASSP) principles and promotes the development of
comprehensive and integrated service delivery systems through a system of care model. Goals of
this initiative are to develop and expand both the interagency infrastructure and the service
delivery system so that a wide array of family-driven and youth-guided individualized services
can be provided to children, youth, and families in an integrated, community-based, and
culturally and linguistically competent manner. The system of care philosophy is
comprehensively described in the seminal 1986 monograph by Beth Stroul and Robert Friedman.
The system of care assessment has three primary goals. First, it provides a description of each
CMHS-funded system to document how system of care communities have operationalized the
system of care principles. Second, it periodically assesses the program’s status in order to track
system development over time. Finally, the system of care assessment enables us to compare
systems on the extent to which they embody system of care principles. These goals are critical to
the advancement of knowledge about systems of care. In essence, they allow us to test the system
of care program model and to document information that can be used to replicate the approaches
that achieve the greatest improvements in child, youth, and family outcomes.

Underlying Framework
The purpose of this framework is to guide the system of care assessment component of the
national evaluation. The wide variation in the way CMHS-funded programs implement their
systems of care requires that this tool be standardized to assess the programs reliably, but
sufficiently flexible to capture the essential features. To accomplish this, a framework was
needed that could be used to 1) describe the basic generic components of any delivery system,
and 2) rate each component on how well it has realized key system of care principles. Following
the literature, the assessment tool and other work done in the field, the framework was divided
into two separate tables, one for each domain: the system infrastructure, and the service delivery
process.
Interpreting the Framework
The columns represent the generic components that can be found in most service systems.
Because good and effective services can be delivered in a variety of ways, it is difficult to
determine whether a given approach to a component of the system is inherently better than
another. For example, system governance can be conducted in many different ways. All
approaches may be equally acceptable and achieve equally successful outcomes. Rather than
valuing (and rating) the approach, for each component a straightforward general description will
be provided. The infrastructure table has four components addressing the general areas of
governance, management and operations, service array, and program evaluation. The four system
components of the service delivery table are entry into services, service planning, service
CMHS National Evaluation
Phase VI February 2011

1

provision, and care review. The components’ definitions, as they are used in this framework are
provided below.

Definition of System Components
Infrastructure
Governance - The governing structure responsible for explicating the system’s goals, vision, and mission,
strategic planning and policy development, and establishing formal arrangements among agencies. This structure
may involve boards of directors, oversight or steering committees, or interagency boards and structures.
Management and operations - The administrative functions and activities that support direct service delivery.
This component of the framework focuses primarily on staff development, funding approaches, and procedural
mechanisms related to the implementation of the service system.
Service array - The range of service and support options available to children and their families through the
system of care.
Program Evaluation – Program evaluation conducted through the integration of process assessment and
outcome measurement, and the use of continuous feedback loops to improve service delivery.
Service Delivery
Entry into service system - The processes and activities associated with the child, youth, and family’s initial
contact with the service system(s) including eligibility determination.
Service planning - The identification of services for the child, youth, and family through an initial process and
periodic updating of service plans.
Service provision - The processes and activities related to the child or youth's on-going receipt of and
participation in services.
Care review – Processes and activities related to the formal review of care of individual children and youth to
address complex issues and challenging problems to prevent the use of more restrictive services or settings.

The rows represent selected system of care principles. According to the program model,
systems of care should be family-driven and youth guided, demonstrate interagency
collaboration, and provide individualized, culturally and linguistically competent, coordinated
and accessible services that are community-based and in least-restrictive environments. In
general, the principles have been defined broadly and applied in the field. For this purpose,
however, it was necessary to develop working definitions of the system of care principles that
were more narrowly construed and that could be made explicit. Definitions of the principles, as
they are operationalized for this study, are provided below.
Each component of the framework will be rated on the extent to which it manifests system of
care principles. In the cells of the table, systems will be rated on how well the component for that
column embodies the principle on that row. Each cell of the framework outlines the indicators
upon which the rating will be based. To make this a practical tool, the indicators of the cells have
been limited to those that were necessary and could be reasonably assessed.
CMHS National Evaluation
Phase VI February 2011

2

Definition of Principles
Family-driven - The recognition that: (1) the ecological context of the family is central to the care of all
children; (2) families are primary decision makers and equal partners in, all efforts to serve children; and (3) all
system and service processes should be planned to maximize family involvement and decision-making.
Individualized - The provision of care that is expressly child- and youth-centered, that addresses the child or
youth’s specific needs and that recognizes and incorporates the child or youth’s strengths.
Youth guided – The recognition that young people have a right to be empowered, educated, and given the
opportunity to make decisions about their own care; and about the policies and procedures governing the care of
all youth.
Culturally and linguistically competent - Sensitivity and responsiveness to, and acknowledgment of, the
inherent value of differences related to race, religion, language, national origin, gender, socio-economic
background and community-specific characteristics.
Interagency - The involvement and partnership of core agencies in multiple child-serving sectors including child
welfare, health, juvenile justice, education, and mental health.
Collaborative/Coordinated - Professionals working together in a complimentary manner to avoid duplication of
services, eliminate gaps in care, and facilitate the child’s and family’s movement through the service system.
Accessible - The minimizing of barriers to services in terms of physical location, convenience of scheduling, and
financial constraints.
Community Based - The provision of services within close geographical proximity to the targeted community.
Least restrictive - The priority that services should be delivered in settings that maximize freedom of choice and
movement, and that present opportunities to interact in normative environments (e.g., school and family).

In developing the System of Care Assessment tool, several steps were taken to maximize
measurement quality. First, the framework was reviewed by experts in the field and revisions
were made. Second, the interviews were developed following closely from the framework. Third,
the interviews were pilot-tested in four sites and revisions were made based on those
experiences. The revised guides were again reviewed by experts. Finally, the tool was applied in
13 funded system of care communities and 3 comparison communities and refined again. Minor
revisions have been made during the decade since it was first developed. Each revision followed
the same process as outlined above: revisions based upon relevant empirical evidence and
literature review; expert review and revision; application in the field with further refinement as
needed. Throughout the process inter-rater reliability was assessed following training for all site
visitors, and in the field among the persons leading the study. In both settings, inter-rater
reliability met or exceeded the threshold of 85 percent agreement.
Indicator Scores
Some of the items in the interviews are for context or descriptive purposes while others are
linked to indicators in the framework. The items that map onto framework indicators are shown
CMHS National Evaluation
Phase VI February 2011

3

on the interview item in parentheses (e.g., B.5.a., where ‘B’ is the column on the framework, ‘5’
is the row on the framework, and ‘a’ is the indicator in that cell). To rate an item, interviewers
use the response provided from the individual respondent to rate the system on a five-point scale
(with 1 being the lowest and 5 being the highest) using the established criteria for that item. That
is, the qualitative data collected in the semi-structured interview are used to rate the system of
care community on each item and the responses of the various stakeholder informants are rated
separately.
For several items in the youth interview (P), caregiver interview (I) and the family representative
interview (C), respondents are asked directly to rate, on a scale from 1 to 5, their experiences
with a given process. In those cases, interviewers also are asked to rate that experience based on
how the respondent described it. This was done to obtain another perspective for items where the
respondent’s appraisal of the experience is the most important, but where research has shown
that reports tend to be overly positive. Having the interviewer also rate the process allows the
examination of discrepancies between respondent and interviewer perspectives.

CMHS National Evaluation
Phase VI February 2011

4

INFRASTRUCTURE DOMAIN
The organizational arrangements and procedural framework that support and facilitate service delivery
A
Governance
Component The governing structure responsible for
explicating the system’s goals, vision, and
mission,
strategic
planning
and policy
development,
and
establishing
formal
arrangements among agencies and family
organizations. These may include boards of
directors,
oversight/steering
committees,
Principle
interagency boards and structures.
1

B
Management and Operations

C
Service Array

The administrative functions and activities that The range of service and support options available
support direct service delivery. This element of to children, youth and their families through the
the framework focuses primarily on staff system.
development, funding approaches, and
procedural mechanisms related to the
implementation of the service system.

a. Family representatives are actively involved a. Direct service staff and their supervisors are
in key governing body functions [Table 1;
trained to provide family-driven care
Family Rep (C7); Core Agency Rep (A10)]
[Table 2]

Family-driven
b. Family representatives are given accurate, b. The staffing structure includes lay-persons
and
paraprofessionals
(e.g.,
family
understandable, and complete information
members)
to
support
families
in
the
care
of
necessary to fulfill their role on the
their
children
[Project
Director
(B23);
governance body [Family Rep (C8)]
Family Rep (C14)]
c. Meetings related to the governance of the
system are held at convenient times and places c. Families are actively involved in grant
operations (e.g., design and implement
to maximize opportunities for family
programs; provide training, serve as staff,
representatives to attend [Family Rep (C10)]
etc.) [Project Director (B30); Family Rep
(C12)]
d. There are mechanisms in place to facilitate

D
Program Evaluation
The process of formal collection, analysis, and
integration of process and outcome data, and the
use of continuous feedback loops to improve
program development, implementation, and
direct service delivery.

a. There are family advocacy, peer support, and a. Information on family outcomes is used to
other support services in the array (e.g., parent
improve services [Evaluator (D9)]
support groups, behavior management training,
empowerment efforts) [Table 4]
b. Information on families’ experiences with the
service delivery system is used to improve the
service system [Evaluator (D10)]
c. Families are involved in the program
evaluation process (e.g., choose indicators to
be monitored, develop focused studies,
participate in data collection process, report
findings to stakeholders) [Evaluator (D5)]

family representative’s participation in
meetings related to the governance of the
system [Family Rep (C11)]

CMHS National Evaluation
System of Care Assessment/Phase VI
Infrastructure Domain, February 2011

1

2
Youth-guided

A
B
C
D
Governance
Management and Operations
Service Array
Program Evaluation
a. Youth are actively involved in key governing a. The staffing structure includes youth (e.g., a. There are youth advocacy, peer support, and a. Youth are involved in the program evaluation
body functions [Core Agency Rep (A11);
volunteer or paid program staff; peer
other support services in the array (e.g., youth
process (e.g., help choose indicators to be
mentors, youth group leaders) [Project
Family Rep (C9); Youth (P21); Youth
support groups, youth empowerment efforts)
monitored, help develop focused studies,
Coordinator (Q4)]
Director (B31); Family Rep (C13); Youth
participate in data collection process, report
[Table 4]
(P25); Youth Coordinator (Q7)]
findings to stakeholders) [Evaluator (D6) ]
b. Youth representatives are given accurate,
understandable, and complete information
b. Information on youth experiences with the
necessary to fulfill their role on the governance
service delivery system is used to improve the
body [Youth (P22)]
service system [Evaluator (D13)]
c. Meetings related to the governance of the
system are held at convenient times and places
for youth representatives to attend [Youth
(P23); Youth Coordinator (Q5)]
d. There are mechanisms in place to facilitate
youth
representative
participation
in
governance activities [Youth (P24); Youth
Coordinator (Q6)]

3
Individualized

CMHS National Evaluation
System of Care Assessment/Phase VI
Infrastructure Domain, February 2011

a. Mechanisms are in place to maximize the a. The service array is complete such that key a. Information on child/youth outcomes is used
provision of individualized care [Project
service options are not missing [Table 4; Care
to improve service delivery [Evaluator (D12)]
Director (B18); Family Rep (C23)]
Coordinator (F26); Other Agency Staff
(L15)]
b. Information on the individualization of
b. Staff receive training on the provision of
services is used to improve service delivery
individualized care [Table 2]
[Evaluator (D11)]

2

A
Governance

4
Cultural and
linguistic
competence

B
Management and Operations
a. Cultural diversity resembling that of the a. Direct service staff, their supervisors
receive training on the provision of
intended service population is evident in the
culturally and linguistically competent care
active and voting membership of key
[Table 2]
governing bodies [Table 1]
b.

c.

Efforts are made to promote the cultural and b. Efforts are made to ensure the cultural and
linguistic competence of the grant program
linguistic competence of program
[Cultural Competence Coordinator (R1)]
management and operations [Cultural
Competence Coordinator (R3)]
Efforts are made to ensure the cultural and
linguistic competence of the governing body
[Cultural Competence Coordinator (R2)]
c. Efforts are made to recruit, hire or contract
with staff and service providers who reflect
the cultural and linguistic background of the
intended population [Project Director
(B21); Cultural Competence Coordinator
(R4)]

C
Service Array

D
Program Evaluation

a. The cultural and linguistic background of the a. Information related to the provision of
intended service population is considered in the
culturally and linguistically competent care is
development of the service array [Project
used to improve service delivery [Evaluator
Director (B19); Family Rep (C20); Cultural
(D14)]
Competence Coordinator (R5)]
b. Mechanisms are in place to ensure that the
program evaluation process is culturally and
linguistically competent [Evaluator (D8)]

d. Efforts are made to accommodate language
preferences of the child, youth and family in
service delivery [Project Director (B22)]

5
Interagency

a. Agencies from the core child-serving sectors a. Shared administrative processes (e.g., a. The array includes services provided by or a. Multiple agencies (across the core child
are actively involved in key governing bodies
shared forms, integrated MIS) facilitate the
through the core child-serving agencies or sectors
serving sectors) are involved in program
and functions [Core Agency Rep (A6, A9);
involvement of the core child-serving
evaluation activities [Evaluator (D7)]
[Table 4]
agencies in grant operations [Project
Table 1]
b. Information
related
to
interagency
Director (B32); Core Agency Rep (A14)]
b. There are structural mechanisms in place to
involvement is used to improve service
maximize
interagency involvement in b. There are mechanisms in place to integrate
delivery [Evaluator (D15)]
governance of the system [Core Agency Rep
staff across agencies (e.g., staff from
various agencies are trained together, co(A8)]
staff, out-posted or co-located) [Project
Director (B24); Core Agency Rep (A13)]
c. Mechanisms are in place to pool or blend
funding across agencies [Project Director
(B33); Core Agency Rep (A15)]
d. Agencies’
routine
operations
are
altered/improved as a result of involvement
in grant [Core Agency Rep (A22)]

CMHS National Evaluation
System of Care Assessment/Phase VI
Infrastructure Domain, February 2011

3

A
Governance
6
Collaborative/
coordinated

B
Management and Operations

C
Service Array

D
Program Evaluation

a. A process is in place to facilitate sharing a. The service array includes service option(s) to a. Information related to the coordination of
information about procedures related to
coordinate services, help families negotiate and
services is used to improve service delivery
grant operations with supervisory and direct
navigate
the
system,
and
facilitate
[Evaluator (D17)]
line staff in agencies, and contract providers
communication among providers and agencies
(e.g.,
case/care
management,
service
[Project Director (B34); Core Agency
coordination function) [Table 4]
Rep (A12); Social Marketing Manager
(S14); Other Agency Staff (L5)]
b. Mechanisms are in place to facilitate the
coordination of services across providers,
agencies and organizations [Project
Director (B12); Core Agency Rep (A16)]

7
Accessible

a. There are mechanisms in place to minimize a. Efforts are made to ensure that services within a. Information related to the accessibility of
financial barriers to services and care
the array have adequate capacity to serve all
services is used to improve service delivery
who need them [Project Director (B13);
[Project Director (B15); Family Rep
[Evaluator (D18)]
(C19)]
Family Rep (C17); Core Agency Rep (A19)]
b. Efforts are made to maximize the accessibility
of the service array [Project Director (B14);
Family Rep (C18); Core Agency Rep (A20)]
a. The full array of services is provided within the a. The use of services provided or located outside
community [Table 4]
the community is monitored and that
information is used to reduce their use
b. Efforts are made to minimize the need for
[Evaluator (D19)]
children/families to leave the community for
services [Project Director (B16); Family Rep
(C21); Core Agency Rep (A17)]

8
Community
based

9
Least
restrictive

CMHS National Evaluation
System of Care Assessment/Phase VI
Infrastructure Domain, February 2011

a. Staff receive training on the use of least
restrictive care [Table 2]
b. Procedures are in place to minimize the
inappropriate use of restrictive service
options [Project Director (B17); Family
Rep (C22); Core Agency Rep (A18)]

a. Information related to the use of overly
restrictive service options is monitored and
that information is used to reduce their use
[Evaluator (D20)]

4

SERVICE DELIVERY DOMAIN
The activities and processes undertaken to provide services to children and families for the purpose of addressing and, to the extent possible, relieving the emotional and behavioral challenges experienced by the child
E
Entry into Service System

F
Service Planning

G
Service Provision

H
Care Review

Component The processes and activities associated with a The process for initial identification of services The processes and activities related to a child or Processes and activities related to the formal
child or youth and family’s initial contact with the and service plan development for a child or youth’s ongoing receipt of and participation in review of care of individual children and youth to
service
system(s)
including
eligibility youth and family.
services.
address complex issues and challenging problems
determination.
to prevent the use of more restrictive services or
settings.
Principle
1
Family-driven

a. Entry into the service system is family friendly a. Families have key decision making roles in a. Families are fully involved in and make a. Families are involved in the care review
the service planning process (they identify
informed decisions about service provision
process for their child, youth and family.
[Family (I7)]
strengths and needs, develop goals and
[Family (I32); Service Provider (G12)]
[Care Review Participant (H9)]
objectives, identify and select team
b. Understandable information about the care
participants, identify and select service b. Services identified and planned with the family
review process and the issues to be discussed
options, reject team members or suggested
are received [Family (I24); Care Coordinator
is provided to the family prior to the meeting
services, etc.) [Family (I11); Care
(F25)]
[Care Review Participant (H10)]
Coordinator (F10); Other Agency Staff
(L13)]
c. The strengths of the family are used to direct the
provision of services [Service Provider (G10)]
b. Family’s strengths and needs are assessed
and services are identified and planned that
strengthen and support the family in the
care of their child or youth [Family (I17);
Care Coordinator (F12)]
c. Providers, care coordinators, and others
involved in service planning, recognize and
use strengths in the family to plan services
[Family (I16); Care Coordinator (F14);
case record review]

CMHS National Evaluation
System of Care Assessment/Phase VI
Service Delivery Domain, February 2011

5

2
Youth-guided

E
F
G
H
Entry into Service System
Service Planning
Service Provision
Care Review
a. Entry into the service system is youth friendly a. Child/youth is fully involved in service a. Youth are fully involved in service provision a. Child/youth is involved in the care review
planning process [Family (I12); Care
[Youth (P8); Youth Coordinator (Q16)]
process [Care Review Participant (H11)]
[Youth (P15)]
Coordinator (F11); Other Agency Staff
b. Understandable information about the
(L14); Youth (P11); Youth Coordinator
care review process and the issues to be
(Q17)]
discussed is provided to the child/youth
prior to the meeting [Care Review
Participant (H12)]
a. Individualized service plans are developed a. Services identified and planned for the child and
for each child and youth in the system
youth are received [Family (I23); Care
[Care Coordinator (F7); Other Agency
Coordinator (F24); Youth (P14)]
Staff (L12)]
b. The strengths of the child and youth are used to
b. The strengths of the child and youth are
shape the provision of services [Service
utilized when planning for services [Family
Provider (G9)]
(I13); Care Coordinator (F13); Youth
(P13); case record review]

3
Individualized

4
Cultural and
linguistic
competence

c. Service plan matches child and youth’s
individual needs [Family I15]
a. There is active outreach to specific cultural a. Families’ culture is routinely assessed and d. Culture of the child, youth and family is used to
incorporated into the service planning
direct service delivery [Service Provider (G11)
groups or populations [Project Director (B9);
process
[Family
(I18);
Care
Coordinator
Social Marketing Manager (S13)]
e. Language preferences of the child, youth and
(F17); case record review]
family can be accommodated in services received
b. Intake is conducted in the preferred language of
b.
Language
preferences
of
the
child,
youth,
[Family (I19b)]
the family [Intake (E7)]
and family can be accommodated in service
planning
[Family
(I19a);
Care
Coordinator (F16)]

CMHS National Evaluation
System of Care Assessment/Phase VI
Service Delivery Domain, February 2011

6

E
Entry into Service System

5
Interagency

6
Collaborative/
Coordinated

7
Accessible

F
Service Planning
a. Referrals come from multiple agencies across a. All involved child-serving agencies
routinely participate in the service planning
child-serving sectors [Intake (E2); Evaluator
process [Family (I9); Care Coordinator
(D16)]
(F18); Other Agency Staff (L9); Youth
(P10)]
b. Multiple agencies across sectors are able to
conduct intake into the grant program [Intake
(E3)]

G
Service Provision

H
Care Review
a Agencies across the child-serving sectors
participate in care review [Care Review
Participant (H14)]

a. There are efforts to inform community-based a. Involved providers and organizations, a. Providers, organizations, and agencies work a. Proceedings, findings, and decisions from care
organizations, private providers, family
routinely participate in the service planning
together to coordinate service provision [Family
review meetings are routinely disseminated
organizations, support groups, etc. about the
process [Care Coordinator (F20); Other
among all involved agencies, providers, and
(I30); Care Coordinator (F35); Service
grant and its services [Project Director
organizations [Care Review Participant
Agency Staff (L10)]
Provider (G13)]
(B10); Social Marketer (S12)]
(H16)]
b. The service planning process (including the
service plan) is coordinated across agencies,
b. Any provider or organization involved in the
organizations, and providers [Family (I10);
child, youth, or family’s care can request a
care review meeting [Care Review
Care Coordinator (F21); Other Agency
Staff (L11)]
Participant (H15)]
a. There is active and ongoing outreach to the a. Service planning meetings occur at flexible a. Services have sufficient capacity to serve all
.
those who need them [Care Coordinator (F27);
intended population [Project Director (B8);
times to maximize the convenience for the
Other Agency Staff (L16)]
child, youth and family [Care Coordinator
Youth
Coordinator
(Q14);
Social
Marketing Manager (S11)]

(F8)]

b. Services are provided at flexible or extended
hours [Family (I25); Care Coordinator (F4);
b. The process to enter the service system is b. Service planning meetings occur at a variety
Service Provider (G6)]
simple and uncomplicated for youth and
of places to maximize the convenience for
families [Intake (E4); Family (I5); Other
the child, youth and family [Care
c. Services are provided in convenient locations
Agency Staff (L6); Youth (P6)]
Coordinator (F9)]
[Family (I26); Care Coordinator (F6); Service
Provider (G8)]
c. The length of time between referral and receipt
of services is minimal [Intake (E5); Family
d. Transportation to services is available [Family
(I6); Other Agency Staff (L7); Youth (P7)]
(I29); Care Coordinator (F32)]
e. Services are financially accessible to families
[Family (I27); Care Coordinator (F31)]
f. Services are accessible in a timely manner (wait
for services is minimal) [Family (I28); Care
Coordinator (F28)]

CMHS National Evaluation
System of Care Assessment/Phase VI
Service Delivery Domain, February 2011

7

8
Community
based
9
Least
restrictive

CMHS National Evaluation
System of Care Assessment/Phase VI
Service Delivery Domain, February 2011

a. Children, youth and families receive services in a. The care review process fully explores
community-based service options to avoid
their home communities [Care Coordinator
children, youth and families from having to
(F30); Other Agency Staff (L17)]
travel out of their home communities for
services [Care Review Participant (H22)]
a. For children and youth being served in restrictive a. The care review process ensures that less
service options (e. g., out of school, out of
restrictive options are exhausted before more
home), efforts are made to use progressively less
restrictive services or placements are
restrictive service options [Care Coordinator
considered [Care Review Participant (H23)]
(F33); Other Agency Staff (L18)]

8

System of Care Assessment Introduction Letters
[Date]
[Project Director’s Name]
[Address]
Dear [Title and Last Name]:
As part of the National Evaluation of the Comprehensive Community Mental Health Services for
Children and Their Families (CMHS) Program, members of the National Evaluation Team will be
conducting site visits every 18–24 months, beginning in 2011, to assess the development and
implementation of your system of mental health care for children and youth and their families. The
assessment process will occur over a 3-day time period within a single week. During that time, a team of
two site visitors will conduct interviews with many persons involved in your project, as well as review a
sample of case records.
We plan to visit your project in [Month] 2011 and would like for you to designate on the enclosed
calendar your first and second choices of weeks and at least 3 consecutive days within those weeks when
you would prefer the visit take place. Please return the calendar by [Date], with your choices marked on
it, to Matosha Glover at [email protected] or to:
ICF Macro
3 Corporate Square, NE, Suite 370
Atlanta, GA 30329
or
(404) 321-3688 (fax)
To assist you in identifying dates for the site visit, we will want to meet with you for the first interview on
the first day and again at the end of the final day for a debriefing session. Interviews also will be
conducted with direct service staff, family caregivers and youth, core agency representatives, family
advocacy group members, therapists, and other community agency staff. We will send to you, at a later
date, instructions and a more specific and detailed site informant list and set of data tables for you to
complete prior to the visit. We will send these forms to you electronically in Microsoft Office Word for
the convenience of electronic completion. As the site visit dates approach, we will confirm with you the
final plans and daily interview schedules for the visit.
If you have any questions or concerns regarding the scheduling process or the site visit, please do not
hesitate to contact me at (404) 321-3211. We look forward to our visit to your project.
Very truly yours,
Freda Brashears
Project Manager
System of Care Assessment Study
Enclosure
cc:

[Evaluator]
[Family Rep]
[CMHS Project Officer]
[Site Liaison]

National Evaluation of the Comprehensive Community Mental Health Services
for Children and their Families Program

System of Care Assessment
Site Visit Preferences
Please complete the identifying information below and select your first and second choices of weeks—with at least 3
consecutive days within those weeks—as your preference for a data collection site visit.

Project Name
Contact Person
Telephone _______________________________ FAX
E-mail
First Choice ________________________ Second Choice

April 2011
S

M

T

W

T

F

S

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

Return by January 12, 2011 by mail or fax to
Matosha Glover
ICF Macro
3 Corporate Square, NE, Suite 370
Atlanta, GA 30329
(404) 321-3211
(404) 321-3688 (FAX)

[Date]
[Project Director’s Name]
[Address]
Dear [Title and Last Name]:
As part of the National Evaluation of the Comprehensive Community Mental Health Services for
Children and Their Families Program, members of the National Evaluation Team will be conducting site
visits every 18–24 months, beginning in 2011, to assess the development and implementation of your
system of mental health care for children and youth and their families. The assessment process will occur
over a 3-day time period within a single week. During that time, a team of two site visitors will conduct
interviews with many persons involved in your project, as well as review a sample of case records.
We plan to visit your project in [Month] 2011. Our site visit team is available to visit your community on
[dates and dates]. Please designate on the enclosed calendar your first and second preferences of these
dates. Please return the calendar by [Date], with your preferences marked on it, to Matosha Glover at:
ICF Macro
3 Corporate Square, NE, Suite 370
Atlanta, GA 30329
or
(404) 321-3688 (fax)
To assist you in selecting your preferred dates for the site visit, we will want to meet with you for the first
interview on the first day and again at the end of the final day for a debriefing session. Interviews also
will be conducted with direct service staff, family caregivers and youth, core agency representatives,
family advocacy group members, therapists, and other community agency staff. We will send to you, at a
later date, instructions and a more specific and detailed site informant list and set of data tables for you to
complete prior to the visit. We will send these forms to you electronically in Microsoft Office Word for
the convenience of electronic completion. As the site visit dates approach, we will confirm with you the
final plans and daily interview schedules for the visit.
If you have any questions or concerns regarding the scheduling process or the site visit, please do not
hesitate to contact me at (404) 321-3211. We look forward to our visit to your project.
Very truly yours,
Freda Brashears
Project Manager
System of Care Assessment Study
Enclosure
cc:

[Evaluator]
[Family Rep]
[CMHS Project Officer]
[Site Liaison]

National Evaluation of the Comprehensive Community Mental Health Services
for Children and their Families Program

System of Care Assessment
Site Visit Preferences
Please complete the identifying information below and select your first and second choices of weeks—with at least 3
consecutive days within those weeks—as your preference for a data collection site visit.

Project Name
Contact Person
Telephone _______________________________ FAX
E-mail
First Choice ________________________ Second Choice

April 2011
S

M

T

W

T

F

S

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

Return by January 12, 2011 by mail or fax to
Matosha Glover
ICF Macro
3 Corporate Square, NE, Suite 370
Atlanta, GA 30329
(404) 321-3211
(404) 321-3688 (FAX)

[Date]

[Project Director’s Name]
[Address]
Dear [Title and Last Name]:
It is time once again for us to schedule our visit to your Children’s Mental Health Initiative project as part
of the national evaluation. The purpose of our visit is to assess the development and implementation of
your system of care. As you recall from our previous visit, the assessment occurs over a 3-day time period
within a single week. During that time, a team of two site visitors will conduct interviews with many
persons involved in your project as well as review a sample of case records.
We plan to visit your project in [Month] 2012 and would like for you to designate on the enclosed
calendar your first and second choices of weeks and at least three (3) consecutive days within those weeks
when you would prefer the visit take place. If your Federal SAMHSA visit is also scheduled for this
month and you would like to arrange for both visits to occur in the same week, please indicate so on your
response. Please return the calendar by [Date], with your choices marked on it, to Matosha Glover at:
ICF Macro
3 Corporate Square, NE, Suite 370
Atlanta, GA 30329
or
(404) 321-3688 (fax)
As a reminder for your consideration in identifying dates for the site visit, we will want to meet with you
for the first interview on the first day and again at the end of the final day for a debriefing session.
Interviews will also be conducted with direct service staff, family caregivers and youth, core agency
representatives, family advocacy group members, therapists, and other community agency staff. We will
send to you at a later date the site informant list and set of data tables for you to complete prior to the
visit. We can send these forms to you electronically in MSWord for the convenience of online
completion. As the site visit date approaches, we will confirm with you the final plans and daily agenda
for the visit.
If you have any questions or concerns regarding the scheduling process or the visit itself, please do not
hesitate to contact me at (404) 321-3211. We look forward to our visit to your project.
Very truly yours,

Freda Brashears
Project Manager
Enclosure
cc:

[Evaluator]
[Family Rep]
[CMHS Project Officer]
[Site Liaison]

National Evaluation of the Comprehensive Community Mental Health Services
for Children and their Families Program

System of Care Assessment
Site Visit Preferences
Please complete the identifying information below and select your first and second choices of weeks—with at least 3
consecutive days within those weeks—as your preference for a data collection site visit.

Project Name
Contact Person
Telephone _______________________________ FAX
E-mail
First Choice ________________________ Second Choice

April 2012
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

Return by January 12, 2012 by mail or fax to
Matosha Glover
ICF Macro
3 Corporate Square, NE, Suite 370
Atlanta, GA 30329
(404) 321-3211
(404) 321-3688 (FAX)

System of Care Assessment Confirmation Letter
[Date]

[Project Director’s Name]
[Address]
Dear [Title and Last Name]:
I am writing to confirm the dates for the National Evaluation of the Comprehensive Community Mental
Health Services for Children and their Families Program site visit to assess your program’s system of
care development and to provide other information regarding that work. The visit will take place [Date],
and the site visitors will be [Name] and [Name]. One of these site visitors will be in touch with you
before the visit to discuss final details and arrangements.
In preparation for the visit, enclosed is a set of six data tables, a site informant list, and instructions for
completing them. Also included is a sample agenda to assist in the development of your site visit. For the
convenience of completing and submitting these documents electronically, we are sending these forms
and instructions to you via e-mail. The instructions are fairly comprehensive and should help you in the
completion of these materials.
Tables 1–6
Please refer to the instructions in this mailing to assist in the completion of the tables. The information to
be provided in the tables will assist you in identifying potential respondents for the System of Care
Assessment study, and for whom interviews could be scheduled. Please complete these preliminary tables
as a first step in your planning process and return the tables to us with your Site Informant List by
[Month/Day - 4 weeks prior to site visit].
Site Informant List
The site informant list identifies 7 categories of respondents who offer a variety of perspectives about
your project’s system of care. We need to interview several persons within each category, as indicated on
the form, and ask that you identify potential respondents by name and agency affiliation. We need to
review your projected list of interviewees prior to the final scheduling of interviews to ensure that each
category of respondents is represented adequately. Therefore, please return the preliminary list of
potential respondents, along with the tables mentioned above, to us by [Month/Day - 4 weeks prior to
site visit].
Agenda
The site informant list indicates the number of persons we need to interview for each category and the
time required for each interview. The length of time indicated covers only the actual interview and not
travel and set-up time. Therefore, when developing the daily schedules, please allow extra time for travel
to interview locations, as well as about 10–15 minutes between interviews to allow preparation time for
the site visitors.
All interviews must be conducted with respondents on an individual basis. We are not able to conduct
interviews in conjunction with meals; however, we are available for evening hours and for home- and/or
community-based interviews with families and/or service providers. Concurrent interviews should be
scheduled for the site visit team throughout each day, except for the project director’s interview, which
will be conducted by both site visitors together.

Case Record Review
In addition to the interviews, our site visitors will review a randomly selected sample of case records of
children enrolled in your CMHS program. When developing the daily interview schedule, please allow
each site visitor 2 hours for this activity. Please refer to the instructions for the case record selection
process and timeframes.
Family Caregiver and Youth Stipends
The family caregivers listed on line #7 of the site informant list will receive $25 cash stipends from the
site visitors to help offset their expenses. Youth respondents will receive $15 cash stipends from the site
visitors to thank them for their participation. Stipends will be provided to informants from this category
only.
Debriefing
At the close of the 3-day visit, the site visit team will be available for a joint debriefing session with you,
the program evaluator, and the family organization representative. The purpose of the debriefing is to
bring closure to the visit by providing preliminary feedback and discussing next steps. The site visitors
will not be prepared to present findings during this session as the data analysis will not yet have occurred.
However, within approximately 8 weeks of the site visit we will send a draft report of the findings from
our assessment for your review and comment.
Timeline
The timeline for these preparatory steps is as follows:





Preliminary tables 1–6 completed and returned to us by [Month/Day - 4 weeks prior to site
visit]
Preliminary site informant list completed and returned to us by [Month/Day - 4 weeks prior to
site visit]
We will respond to refine the site informant list by [Month/Day - 2 days after list return date]
Final site informant list, agendas, and six data tables completed and returned to us by
[Month/Day - 2 weeks prior to site visit]

Submission of Materials
Materials sent by e-mail are preferred. Please send all materials to Freda Brashears at
[email protected] or via mail or fax to
Freda Brashears
ICF Macro
3 Corporate Square, NE, Suite 370
Atlanta, GA 30329
or
(404) 321-3688 (fax)
Thank you for assisting us in completing this part of the national evaluation. We look forward to our visit
to your project. Please do not hesitate to contact me at (404) 321-3211, or via e-mail, if you have
questions or concerns about this process.
Very truly yours,
Freda Brashears
Project Manager
Enclosures

cc:

[Evaluator]
[Family Rep]
[CMHS Project Officer]
[Site Liaison]

System of Care Assessment Draft Report Letter
[DATE]

[NAME & ADDRESS]
Dear [Title and Last Name]:
Enclosed is a draft report based on the system of care assessment site visit conducted in [MONTH] and a
summary of your preliminary assessment scores. Also included is background information on the purpose
of the assessment, how the scores are obtained, and some guidance for interpreting scores.
Before finalizing this report, we would like you to review it to make sure that information such as
demographic characteristics of the population served, dates, names of agencies, partners, etc. is correct.
Please make any corrections or edits on hard copy and return them by mail, marked to my attention; or
use Track Changes in the electronic version, and return to me by e-mail ([email protected]). If more
than one person reviews this draft, please combine all of the edits into one document before returning
your comments to us. To ensure accuracy, we will not be able to take any comments or edits over the
telephone.
Please provide your written edits by [3 WEEKS]. After we review your comments, we will make the
appropriate revisions and send you the final report. At that time we will also send a copy to SAMHSA.
Thank you for all of your help with this process. We look forward to hearing from you. If you have any
questions about your scores, please do not hesitate to call me at (404) 321-3211.
Sincerely,

Freda Brashears
Project Manager
Enclosures
cc:

[Evaluator]
[Family Rep]

System of Care Assessment Final Report Letter
DATE

PD’s NAME & ADDRESS
Dear [Title and Last Name]:
Thank you for the thoughtful comments on the draft system of care assessment report for [PROJECT
NAME]. The enclosed final report incorporates all comments that corrected the factual information
presented or provided additional clarification. [Those comments that reflected a perspective that was
different from what we learned on-site from interviews with multiple respondents or reflect changes that
have been made since the site visit have not been incorporated.]
The assessment scores and narrative reports are important sources of information for the national
evaluation. They allow us to examine trends in system of care development over time. Many grant
communities have indicated that they have found their reports useful for program development, strategic
planning, partnership building, decision making, and other activities. We hope your report provides
similar benefits to you as you continue your system development and sustainability efforts.
A set of all final site visit reports will be sent to the Center for Mental Health Services. Other partners in
the services program will also receive a set. These partners include the National Federation of Families
for Children’s Mental Health, Technical Assistance Partnership for Child and Family Mental Health, and
the National Technical Assistance Center for Children’s Mental Health at Georgetown University.
The national evaluation team greatly appreciates your efforts during the site visit process, including
providing requested documentation, completing forms, scheduling the interviews, and, especially, setting
aside staff and family time to meet with our site visitors. We also appreciate the time you have taken to
review and comment on the draft report. We hope this has been a productive and positive experience.
Sincerely,
Freda Brashears
Project Manager
Enclosure
cc:

Evaluator (w/enclosure)
Family Organization Rep. (w/enclosure)
Ingrid Goldstrom (w/enclosure)
CMHS Project Officer (w/enclosure)
TA Partnership (w/enclosure)
TAC (w/enclosure)
both site visitors (w/enclosure)
site liaison

System of Care Assessment Thank You Letter

[Date]

[Project Director Name & Address]
Dear [Title and Last Name]:
I want to formally thank you and (name) for your participation in our recent system of care
assessment visit. We appreciate the effort it took to plan and organize the visit and thank all of
you for the kindness and hospitality shown our site visitors. We are especially grateful for the
time given by families, staff, and administrators to this data collection effort and ask that you
pass on to them our thanks.
A draft report of the site visit will be sent to you in the next few weeks for your review and
comment. Please feel free to contact me if there are questions or concerns.
Again, thank you for assisting us in accomplishing this part of the national evaluation.
Very truly yours,

Freda Brashears
Project Manager

cc:

(name)

System of Care Assessment Table of Informants and Corresponding Interview Guides
Informant

Representatives of Core Child Serving
Agencies
Project Director
Family Representative to the Governance
Structure
Family Representative to Family
Organization
Program Evaluator
Family Rep to Program Evaluation
Intake Staff
Case Management Staff
(also called Care Coordinators)
Direct Service Providers
(those employed by or through the grant;
those from other agencies who work with
children served by the grant)
Care Review Participants
Caregivers of children receiving services
through the grant
Other Agency Direct Service Staff
(staff from other child-serving
sectors/agencies involved with
children/families also served by the grant
Review of case records

Function/Topic Covered

Governance
Governance
Management and Operations
Governance
Management and Operations
Program Evaluation
Governance
Management and Operations
Program Evaluation
Program Evaluation
Program Evaluation
Entry into the Service System
Service Planning
Service Provision
Service Array
Service Planning
Service Provision
Service Array

Care Review
Service Entry
Service Planning
Service Provision
Service Planning
Service Provision
Service Array

Service Planning
Service Provision

Other Staff/Interviewees(any person
interviewed who does not meet above
descriptions)
Debrief Guide
Youth Participant

Youth coordinator
Cultural and Linguistic Competence
Coordinator
Social Marketing Manager/Coordinator

Debriefing Information for Site
Governance activities
Service Planning
Service Provision
Youth involvement in system of care
activities
Cultural and Linguistic Competence
Activities
Social Marketing Activities

Interview
Guide

Average
number of
informants

Average time required
per interview1

A

3

60 minutes

B

1

120 minutes

C

1

90 minutes

C

1

90 minutes

D

1

45 minutes

D

1

45 minutes

E

1

30 minutes

F

3

120 minutes

G

4

45 minutes

H

2

60 minutes

I

3

90 minutes

L

2

60 minutes

M

Completed by
site visitors

N/A

N

Varies

N/A (used in lieu of other
guides)

O

Presentation by
site visit

N/A

P

2

45 minutes

Q

1

45 minutes

R

1

45 minutes

S

1

45 minutes

1. Interview times vary by respondents; some portions of some guides are not applicable to all respondents.

CMHS National Evaluation
Phase VI March 2009

System of Care Assessment Instructions for Completing Site Visit Tables and Lists

Purpose
The purpose of Tables 1–6 is to acquaint site visitors with the details of what they will see on site.
Because each system of care has a unique set of names, terms, and arrangements specific to its own
community, the information on the Tables gives context to what the visitors see and hear during their
visits.
The information given on the Tables should reflect only the time period covered by the current system of
care assessment site visit. For first-time site visits, the time period would be from the receipt of the grant
until the site visit date. For subsequent visits, the time period would be since the date of the previous visit
to the date of the current visit.
Return the Tables to the national evaluation team along with the Site Informant List.

Table 1
This table shows the breadth and characteristics of the governing body and the extent to which it involves
family, youth, and multi-agency participants.





At the top of the Table give the name of the governing body structure as it should be referenced
during the visit.
List the names of the members of the governing body along with their titles and the agencies they
represent.
Give the demographic information of sex and race/ethnicity of the members.
We will interview 5 people you select from this list. Of the 5, 1 should be a family member and one
should be a youth who serve on the governing body.

Table 2
This table shows the training that has been offered on the various system of care principles and the
breadth of attendance/participation across the local system.





Give the name of the training in the first column.
Give dates the training was held in the second column.
Check the boxes to show representation of attendance from across the system of care.
Interviewees will not be selected from this list.

Table 3
This table presents a big-picture view of the grant-funded program itself. It is a demographics collection
tool and should include all people/positions funded by the grant.



Give staff names, their position/function title, and the demographic information of sex and
race/ethnicity.
It is likely that some of the people on this list will be selected by you for interviews because they will
match the categories of functions given on the site informant list.

CMHS National Evaluation
Phase VI February 2011
1

Table 4
This table is a list of the mandated service categories as given in the authorizing legislation and the
current Guide for Applicants (GFA).




Check the boxes to indicate which of the child- or youth-serving agencies provide the services that
are available in your service array.
List evidence-based treatments available in your service array.
Add all other services in your service array that are available to children and youth and their families
whether or not the grant program provides or funds them and check the boxes to indicate which
child- or youth-serving agencies, private service providers or family organizations provide them.

Table 5
This table shows the amount and sources of program funding and how funds are or are not mixed,
blended, or categorized.






Give the source and amount of funding per source.
Check if the funds are pooled across all elements of the system of care.
Check if the funds are pooled by child-specific case service needs.
Check if the funds are available for categories of services, e.g., transportation, respite, etc.
Answer narrative questions 1 and 2 as applicable.

Table 6
This table is much like Table 1 but refers to the structure (person, committee, or team) that is used by
local system of care communities to review the care of children and youth receiving services through the
grant program to address complex issues and challenging problems to prevent the use of more restrictive
services or settings.




At the top of the Table give the name of the care review structure or team as it should be referenced
during the visit.
List the names of the members of the group along with their titles and the agencies they represent.
Give the demographic information of sex and race/ethnicity of the members.

Site Informant List
The purpose of the site informant list is to identify people who represent the categories of program
functions that we are interested in learning about. From this list of informants, the number and schedule
of interviews is crafted for the site visitors to follow during the visit.





Give names and agency affiliation as indicated for each of the 7 categories listed.
Return the form to the national evaluation team by e-mail for review. We will review the list with
local system of care community staff by telephone to ensure that appropriate people have been
identified for the interviews.
After the telephone review, make any needed revisions.
Return the final version of the form with the interview schedule agenda to the national evaluation
team 2 weeks prior to site visit dates. Materials returned by e-mail are preferred.

CMHS National Evaluation
Phase VI February 2011

Case Record Review
The purpose of the case record review is to use case records of children and youth receiving services
through the CMHS funded program as another source of information regarding the development of the
local system of care. The review does not gather individual child or family names or any other identifying
information, does not document child problems or outcomes, and is not an audit of the interventions used
or any other accountability issue.





Two weeks prior to the site visit please send to the national evaluation team a list of case
identification numbers for records of children and youth who have received services during the
review period and for which you have consent to release them for administrative chart review.
o For first-time site visits, that list will include identification numbers of all children who
have receive services since the receipt of grant funds and for whom you have consent.
o For subsequent visits, the list will include identification numbers of all children who have
received services during the time between the previous site visit and the current site visit
date and for whom you have consent.
The national evaluation team will select a random sample of cases to be reviewed.
The national evaluation team will send the list to the local system of care community by e-mail in
sufficient time for the cases to be pulled and made ready for the site visit.

Interview Schedule/Agenda
The purpose of the interview schedule is to organize and schedule the selected people who will be
interviewed by the site visit team. There will be a team of 2 visitors per site visit. The project director
interview should be scheduled as the first interview of the visit and will be conducted by both site visitors.
Except as directed by the national evaluation team, all subsequent interviews must be scheduled
concurrently for each of the two visitors throughout the remainder of the visit.








Using the site informant list as a guide for numbers of people per category and length of time per
interview, local grant community staff will complete the interview schedule/agenda according to
the availability of the interviewees.
Begin the first day of the interview schedule with the Project Director interview and end the last day
of the interview schedule with the Debriefing.
Leave 10–15 minutes between interviews to allow for set-up time for each new interview.
Visitors are available to travel to various locations, including family homes to complete interviews.
If travel is indicated, include adequate travel time on the agenda.
Include 2 hours for each visitor for case record review.
Interviews should not take place as part of meal times.

CMHS National Evaluation
Phase VI February 2011
3

System of Care Assessment Site Visit Tables

NOTE TO OMB REVIEWER:
No burden is calculated for the completion of these tables. An individual employed by the
program completes this task. Assistance to the national evaluation is consistent with their award
requirements.

CMHS National Evaluation
Phase VI February 2011

Table 1
Participant List for Governing Council
Project Name: _____________________

Governing Council Name: ______________________

Project Location: ___________________

Date of site visit: ________________________________

Name

1

Agency/Organization Affiliation

Race
Please list all that apply.
1=American Indian or Alaska Native
2=Asian
3=Black or African American
4=Native Hawaiian or Other Pacific Islander
5=White

Title/Position

Sex

2

Ethnicity Codes

1=Hispanic/Latino origin
2=Not Hispanic/Latino origin

CMHS National Evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program
Revised 04-11

Race1

Ethnicity2

Table 2
Staff Training Activities
Project Name: ______________________

Project Location: ______________________

Date of site visit: ______________________
Check box if staff from each agency/organization attended that training
Topics

Date(s)

Grant
staff

MH

JJ

ED

CW

Family-driven care
1.
2.
3.
4.
Individualized/Youth-guided care
1.
2.
3.
4.
Cultural and linguistic competence
1.
2.
3.
4.
Least restrictive care
1.
2.
3.
4.
Other (specify)
1.
2.
3.
4

CMHS National Evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program
Revised 01-07

Private
Provider

Family

Other

Table 3
Grant-funded Staff
Project Name: ______________________

Project Location: ___________________

Date of site visit: ____________________
Name

1

Race
Please list all that apply.
1=American Indian or Alaska Native
2=Asian
3=Black or African American
4=Native Hawaiian or Other Pacific Islander
5=White

Function/Position

Sex

2

Ethnicity Codes

1=Hispanic/Latino origin
2=Not Hispanic/Latino origin

CMHS National Evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program
Revised 04-11

Race1

Ethnicity2

Table 4
Summary of Service Array
Project Name: ______________________

Project Location: ___________________

Date of site visit: ____________________
Check which agencies/organizations provide each service
Type of Service

Grant
staff

MH

JJ

ED

CW

Private
Provider

Diagnostic and evaluation services
Neurological and/or neuropsychological assessment
Outpatient individual counseling
Outpatient group counseling
Outpatient family counseling
Medication management
Care management/coordination
Respite care
Professional consultation
24-hour, 7-day-a-week emergency
services, including mobile crisis
outreach and crisis intervention
Intensive day treatment services
Therapeutic foster care
Therapeutic group home
Intensive home-based services (e.g.,
family preservation services)
Transition-to-adult services
Family advocacy and peer support
Residential treatment
Inpatient hospitalization
Primary health care (physical health)
Alcohol and Drug Prevention
Alcohol and Drug Treatment
Evidence-based treatment (EBT):
Other: (add lines as needed)

CMHS National Evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program
Revised 01-07

Family

Other

Table 5
System of Care Funding
Project Name: ______________________

Project Location ___________________

Date of site visit: ____________________
Are these funds:
Amount for the current
fiscal year

Type/source of funds
CMHS grant

pooled across system
of care?

pooled by
case?

categorical?

$

Other Public Funding (e.g., Medicaid, State monies, funding provided through other agencies, additional Federal funding, etc.)
$
$
$
$
$
Private Funding (e.g., support from private foundations, contributions from fund-raising efforts, private insurance dollars, client
co-payments, etc.)
$
$
$
$
Totals

$

1.

Describe the contributions made by partner agencies, in the past year, that are not included in the table.

2.

Describe any flexible funding budgets and how these monies are accessed.

CMHS National Evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program
Revised 04-11

Table 6
Participant List for Care Review Committee/Team
Project Name: ______________________

Care Review Team Name: ______________________

Project Location: ___________________

Date of site visit: ________________________________

Name

1

Agency/Organization Affiliation

Race
Please list all that apply.
1=American Indian or Alaska Native
2=Asian
3=Black or African American
4=Native Hawaiian or Other Pacific Islander
5=White

Title/Position

Sex

2

Ethnicity Codes

1=Hispanic/Latino origin
2=Not Hispanic/Latino origin

CMHS National Evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program
Revised 01-07

Race1

Ethnicity2

Site Informant List
Project Name:

Project Location:

Date of Site Visit:
A

B

C

D

E

Position/Role

Specialized Functions

# of
Interviewees

Time
Required

Names of Participants
(To be filled in by site)

1. Governance Body
Representatives of the core agencies,
family members, and youth involved
in governance of the children’s mental
health service delivery system being
assessed

Please include agency affiliation

Core agency representatives (representatives from
schools, child welfare, juvenile justice, public health,
primary health, etc.) on the governing body.

1.1
Please include agency affiliation

3

60 mins each

1.2
Please include agency affiliation

1.3

2. Project Management and
Operations

CMHS National Evaluation
Phase VII February 2011\

Family member representative on the governing body

1

90 mins

1.4

Youth representative on the governing body

1

45 mins

1.5

Project Director

1

2 hours

2.1

Family Organization Representative

1

90 mins

2.2

Youth Coordinator

1

45 mins

2.3

Cultural and Linguistic Competence Coordinator

1

45 mins

2.4

Social Marketing Manager/Coordinator

1

45 mins

2.5

A

B

C

Position/Role

Specialized Functions

# of
Interviewees

D
Time
Required

1

60 mins

3.1

1

60 mins

3.2

3. Program Evaluation

Lead evaluator for national and local evaluation
efforts
Family representative involved in evaluation

4. Care Review

E
Names of Participants
(To be filled in by site)

Please include staff title or function

Activities related to care review

4.1
2

60 mins each

Please include staff title or function

4.2
5. Service Delivery Staff
These should be direct service staff who
provide services to children, youth, and
families who are part of the CMHS
grant-funded project.

Staff who perform intake

Please include staff title or function

1

30 mins

5.1
Please include staff title or function

Case Management/Care Coordination Staff
5.2
3

Please include staff title or function

2 hours each
5.3

Please include staff title or function

5.4
Please include staff title or function

Therapist/clinician
2

60 mins each

5.5
Please include staff title or function

5.6
Please include staff title or function

Other service staff (e.g., respite provider, mentor,
behavioral aide, family advocate)

5.7
2

60 mins each

Please include staff title or function

5.8

CMHS National Evaluation
Phase VI February 2011

A

B

Position/Role

Specialized Functions

6. Other Agency Direct Service
Delivery Staff
These should be front-line staff from
public child-serving agencies who
provide services to children, youth and
families served by the grant AND who
have attended child and family team
meetings.

C
# of
Interviewees

Please include staff title or function

Staff from other agencies
(e.g.: a teacher or therapist from the schools, a
probation officer, a case worker at child welfare)
who work with children, youth and families served by
the grant

E
Names of Participants
(To be filled in by site)

D
Time
Required

6.1

2

60 mins each

Please include staff title or function

6.2

7. Caregivers and Youth
currently being served by the
project

Caregiver whose child or youth and family has
received services and is a member of a minority
racial or ethnic group

1

90 mins

7.1

These should be family members who
DO NOT serve in staff or advocacy
functions.

Caregiver whose child or youth and family has been
in services for 9 to 12 months

1

90 mins

7.2

Caregiver whose child or youth and family has been
in services for 3 to 6 months

1

90 mins

7.3

1

45 mins

7.4

Youth who receives services and has participated in
his or her own service planning
[Must be 14 years old or older and have parental
consent if under 18 years old]

8. Debriefing

Please include staff title or function

Project Director, Principal Investigator, Director of
Family Organization, Evaluator, others as desired

Please include staff title or function

45 mins
Please include staff title or function

Please include staff title or function

CMHS National Evaluation
Phase VII February 2011\

Sample System of Care Assessment Site Visit Agenda
Site/Project Name
Dates of Site Visit
[Example of 1st Day of a Site Visit]
Wed.
3/31/10
Time

Site Visitor 1

Wed.
3/31/10
Time

Informant

Title or Function

Location

8:30-9:00

Dr. Phil Smith

9:00-11:00

Dr. Phil Smith

project director,
opening/briefing
meeting
project director

11:00-11:15

Informant

Title or Function

Location

124 Green St.,
Room 208

8:30-9:00

Dr. Phil Smith

124 Green St.,
Room 208

124 Green St.,
Room 208

9:00-11:00

Dr. Phil Smith

project director,
opening/briefing
meeting
project director

Break

11:00-11:15

Break

11:15-11:30

travel to next interview (approx. 15 minutes)

11:15-12:15

Program evaluator

124 Green St.,
Room 208

11:30-12:30

Mr. Louis
Fontaine

12:15-1:00

Ms. Angela
Johnson
Lunch

12:30-1:00

Lunch

1:00-2:00

Dr. Pat Malley

therapist

124 Green St.,
Room 210

1:00-1:30

Travel to next interview (approx. 30 minutes)

2:00-2:15

Break

1:30-3:00

Ms. Ilene Barter

2:15-3:15

Dr. Gail Acker

respite coordinator

124 Green St.,
Room 208

3:00-3:15

Break

3:15-3:30

Break

3:15-3:45

travel time to next interview (approx. 30 minutes)

3:30-5:30

Case Record Review

3:45-5:15

Ms. Yolanda
Keith

CMHS National Evaluation
Phase VI December 2009

school system
representative on
governing body

family member in
services for 9 to
12 months

family member in
services for 3 to 6
months

246 Grant St.

458 Piedmont
Ave.

782 Buford Ln

Site Visitor 2

124 Green St.,
Room 208

124 Green St.

Checklist of Planning Steps
Complete Tables 1-6
Use Tables 1 and 6 to direct identification of Governing Body and Case
Review interview respondents
Send preliminary Tables and Site Informant List to the national
evaluation team for review
Schedule the interviews after review with the national evaluation team
Send list of case record identification numbers to the national evaluation
team for chart review sample selection
Interview Scheduling Checklist
Schedule ½ hour at beginning of first day for introductions and
preliminaries
Schedule the Project Director for the first interview on the first day, with
both site visitors together at the same time
Schedule 2 hours for each interviewer to conduct chart review
Have a resource person available to assist with chart review as needed
(explain chart set-up, etc.)
Leave 10-15 minutes between every interview for adequate completion
and set-up time
Leave adequate time for travel between interview locations and for setup time after arrival at destination
Leave 30 minutes between the time of the last interview and the debriefing for site visitor planning time
Do not schedule interviews as part of meals or in groups of respondents

Revised 12-10

System/Program__________________________

Interviewer___________________

Interviewed______________________________

Assessment #_________________

INFORMED CONSENT
System of Care Assessment
Staff
The Center for Mental Health Services in the United States Department of Health and Human Services is
sponsoring a national evaluation of children’s mental health services and systems of care. You are invited to
participate in this evaluation because your community has received funding to improve community-based
mental health services for children and families. Your input is important to helping us understand how
systems of care serve children and what works best. We are asking you to participate in a ______ hour faceto-face interview with a trained interviewer who will ask you to respond to a set of questions about the
children’s mental health system of care in your community. These same questions are asked of other
evaluation participants who perform similar functions in their communities. Here are some things we want
you to know about participating in the interview:
•

Participation in the interview is completely voluntary.

•

You may choose to discontinue the interview at any time, for any reason.

•

Your name will not be used in any reports about this interview and no quotes will be attributed to you.

•

There will be no direct benefit to you from participating in this evaluation. The risk may be the
discomfort some people feel when expressing their opinions or talking about their experiences.

•

A report that combines what we learn from all of the interviews conducted in your community will be
sent to the children’s mental health services program director and other program partners. They may
share that report with others at their discretion.

•

To help keep information about you confidential, we have obtained a Certificate of Confidentiality from
the U.S. Department of Health and Human Services (DHHS). This Certificate adds special protection
for the research information about you. This Certificate does not imply that the Secretary, DHHS,
approves or disapproves of the project. The Certificate of Confidentiality will protect the investigators
from being forced, even under a court order or subpoena, to release information that could identify you.
We may release identifying information in some circumstances, however. For example, we may
disclose medical information in cases of medical necessity, or take steps (including notifying
authorities) to protect you or someone else from serious harm, including child abuse/neglect. Also,
because this research is sponsored by DHHS, staff from DHHS may review records that identify you
during an audit.

•

Any questions you have about the evaluation will be answered before the interview begins.

•

Any questions you may have after the community visit is concluded may be directed to Freda Brashears
at ICF Macro, Atlanta, GA (404) 321-3211.

•

Your signature below indicates that you understand the above and agree to participate.

Participant Printed Name _______________________________________________________
Participant Signature _______________________________________________________________
Witness_________________________________________________________Date______________

CMHS National Evaluation
Phase VII February 2011

2 signed forms: 1 for the interview participant and 1 for the interviewer

Attachment E: Sector and Comparison Study

EDUCATION SECTOR AND COMPARISON STUDY
INFORMED CONSENT—SCHOOL REPRESENTATIVE
Purpose of the Survey

The Center for Mental Health Services (CMHS), a center in the Substance Abuse and Mental
Health Services Administration, an agency within the United States Department of Health and
Human Services, is studying system of care programs. and other agencies and organizations that
provide and/or refer youth to mental health services. You are invited to participate in this interview
because you are working with a youth who is receiving mental health services and also participating in
the evaluation of these services. Your input is important to helping us understand how programs and
agencies serve children and what works best. We are asking you to participate in a 30-minute interview to
respond to a set of questions about the youth(s) with whom you are working and the services provided by
your school. Here are some things we want you to know about participating in the interview:


Participation in the interview is completely voluntary.



You may choose to discontinue the interview at any time, for any reason.



Any information that you provide will be kept strictly private. No one other than study staff will
know who you are or know what answers you gave. Any reports from this survey will report results in
group form. Your name will not be used in any reports about this survey, and no quotes will be used
that would identify you individually.



Your name will not be used in any reports about this interview and no quotes will be attributed to
you.



There will be no direct benefit to you from participating in this evaluation. The risk may be the
discomfort some people feel when expressing their opinions or talking about their experiences.



A report that combines what we learn from all of the interviews conducted in your community will be
sent to the children’s mental health services program director and other program partners. They may
share that report with others at their discretion.



Any questions you have about the evaluation will be answered before the interview begins.



You or your school will be provided an incentive equal to twenty dollars for this interview.



Any questions you may have after the interview is concluded may be directed to [Staff Name] at ICF
Macro at (Research Coordinator phone number inserted here).



Your signature below indicates that you understand the above and agree to participate.

Participant Printed Name ___________________________________________________________
Participant Signature ___________________________________________________________________

Witness _______________________________________________

Date: ___/___/____

JUVENILE JUSTICE SECTOR AND COMPARISON
STUDY
INFORMED CONSENT—AGENCY REPRESENTATIVE
Purpose of the Survey
The Center for Mental Health Services (CMHS), a center in the Substance Abuse and Mental Health
Services Administration, an agency within the United States Department of Health and Human Services,
is studying system of care programs. and other agencies and organizations that provide and/or refer youth
to mental health services. You are invited to participate in this interview because you are working with a
youth who is receiving, or is eligible to receive, mental health services and also participating in the
evaluation of these services. Your input is important to helping us understand how programs and agencies
serve children and what works best. We are asking you to participate in a 30-minute interview to respond
to a set of questions about the youth(s) with whom you are working and the services provided by your
agency/organization. Here are some things we want you to know about participating in the interview:


Participation in the interview is completely voluntary.



You may choose to discontinue the interview at any time, for any reason.



Any information that you provide will be kept strictly private. No one other than study staff will
know who you are or know what answers you gave. Any reports from this survey will report results in
group form. Your name will not be used in any reports about this survey, and no quotes will be used
that would identify you individually.



Your name will not be used in any reports about this interview and no quotes will be attributed to
you.



There will be no direct benefit to you from participating in this evaluation. The risk may be the
discomfort some people feel when expressing their opinions or talking about their experiences.



A report that combines what we learn from all of the interviews conducted in your community will be
sent to the children’s mental health services program director and other program partners. They may
share that report with others at their discretion.



Any questions you have about the evaluation will be answered before the interview begins.



You or your agency will be provided an incentive equal to twenty dollars for this interview.



Any questions you may have after the interview is concluded may be directed to [Staff Name] at ICF
Macro, Atlanta, GA (404) 321-3211.



Your signature below indicates that you understand the above and agree to participate.

Participant Printed Name ___________________________________________________________
Participant Signature ___________________________________________________________________

Witness _______________________________________________

Date: ___/___/____

CHILD WELFARE SECTOR AND COMPARISON STUDY
INFORMED CONSENT—CHILD WELFARE ADMINISTRATOR FOR
CHILDREN IN FOSTER CARE
Purpose
The Center for Mental Health Services (CMHS), a center in the Substance Abuse and Mental
Health Services Administration, an agency within the United States Department of Health and
Human Services, is studying system of care programs. These programs are funded throughout
the country to improve mental health services for children and families. CMHS wants to know
more about these services and how well they work. In order to assess the outcomes of these
programs, ICF Macro and Walter R. McDonald & Associates, Inc. are conducting a Sector and
Comparison Study. The study team is talking to families involved with child welfare who are
receiving services from funded system of care communities and to families involved with child
welfare in communities not receiving funding to develop system of care services. In this study,
we are interested in finding out about the foster child’s behavior and functioning, the kinds of
services the child receive, and how the foster parent feels about these services. The (child
welfare agency name) is a part of this study.
Description of Screening Interview
As part of the study, a member of the study team will interview the person who has been the
primary caregiver of the child over the past 6 months. The primary caregiver is the person who
has had the most interaction with the child in a caregiving role during the previous 6 months. If
the child’s foster parent has been the primary caregiver for the previous 6 months, with your
permission, a member of the study team will interview the foster parent. The screening interview
should take about 25 minutes. In the screening interview, the foster parent will be asked about
the child’s behavior. A member of the study team will talk with the child’s foster parent at home,
or any other place that is best for them.
Description of Interview Participation
If the child is determined to be eligible for the study through the screening interview, a member
of the study team will contact the foster parent to schedule a meeting to obtain consent to
participate in the study. Once consent to participate in the study has been obtained from the
foster parent, a member of the study team will interview the child’s foster parent within the
month following the screening interview and possibly every 6 months for up to 2 years. The
child’s foster parent will be interviewed up to five times in a 2-year period. The child’s foster
parent may be interviewed less than five times in a 2-year period, if the child does not remain in
the care of the foster parent for the full study period. Each interview will take about an hour and
a half. A member of the study team will talk with the child’s foster parent at home, or at any
other place that is best for them. In the interviews, they will be asked about the child, and any
services they have received.
Child Welfare Record Review
As part of the study, a member of the study team, with your permission, will review the child’s
child welfare record. The purpose of this review is to identify services received, child and family

background, and information about child welfare goals.
Services and Cost Study
As part of the study, the study team would also like to review records from other agencies that
have provided services to the child for a Services and Costs Study. The goal of the Services and
Costs Study is to learn about the array of services that children and families receive and the costs
of providing those services. The Services and Cost Study also seeks to examine the relationship
among services, service costs, and outcomes for children and their families.
The records that the study team would review might come from Medicaid, mental health service
providers, Head Start, or early intervention service providers. The information the study team
would obtain from these records is as follows:







Child’s birth date
Dates he/she received services
Description of services received
Unit of service (hours, minutes, days, etc.)
Charge and/or adjusted charge for service
Location of service delivery

The study team member will review records from the 12 months prior to participation in the
study and will review records during the study period of 24 months.
Risks and Benefits
There are no risks or benefits associated with participating in the study. The study team hopes
that the information provided will help to improve services for families with young children with
behavioral or emotional difficulties in the future.
Compensation
For the screening interview, the child’s foster parent will be paid $20. For those children enrolled
into the study, the child’s foster parent will be paid $40 for the baseline interview and for each
interview completed at 6, 12, 18, and 24 months and a $50 bonus at the end of the 24-month
period if they complete all 5 interviews. They will be paid in order to compensate them for the
time they give for the interviews.
Protection of Information
The information obtained about the child from interviews with foster parents and the child’s
record reviews will be used for this study. Only authorized people will have access to the
information. None of the interview or record review forms that are used in the study will have
names on them. Names and contact information will be kept separate under lock and key and
only authorized members of the study team will have access to it. The information is saved on
computers with high levels of security. When study results are reported, answers are grouped
with others and reported in summary form. Reports will never mention any information that
could identify the child. In other words, it may be reported that “68% of families who have
young child with behavioral or emotional difficulties feel very stressed.” The information
obtained in the foster parent’s interviews and the child’s record review will be released to the

national evaluation team, consisting of Walter R. McDonald & Associates, Inc. and ICF Macro,
and our funding agency, the Substance Abuse and Mental Health Services Administration.
In order to get the administrative records on the services provided to the foster parent and the
child and their costs, we will need to give identifying information to the organizations from
which the foster parent and child have received services. As indicated in the Services and Costs
Study section above, the records that the study team would review might come from Medicaid,
mental health service providers, Head Start, or early intervention service providers.
The study team has applied for a Confidentiality Certificate from the Federal government to
protect the people who conduct the interviews from being forced, even under a court order or
subpoena, to identify participants. The Confidentiality Certificate does not imply that the
government has approved or disapproved of this study. In addition, the Federal agency funding
this research may see your information if it audits us to ensure that the study team protects the
rights and safety of all participants.
There are two instances when the study team will not be able to keep your information
confidential. If a study team member finds out that you plan to harm yourself or someone else or
if there is alleged or suspected child abuse or neglect, we may report it to local authorities.
Rights Regarding Decision to Participate
If you agree to have this child participate in the study, understand that you can change your mind
at anytime. If you chose for the child not to participate in the study at anytime, any information
collected for the study will be destroyed, if this is what you want.
Contact Information
If you have any questions about this study, you can call (Research Coordinator) to have your
questions answered. You can call him/her collect at (555) 555–5555. To Contact the Institutional
Review Board that reviewed this study, call 1-877-556-2218.
Voluntary Consent
I have read this form or, it has been read to me, and I understand what it says. My questions have
been answered. A copy of this form will be given to me. By signing my name below, I freely
agree to allow a study team member to complete the following as indicated by a check in the
YES box.
1. To have a study team member conduct a screening interview with the child’s foster
parent, to determine if the child is eligible for the study.
Yes
No
2. To have a study team member interview the child’s foster parent within the month
following the screening interview and every 6 months thereafter, for up to 24 months.
Yes

No
3. To have a study team member access the child’s child welfare agency records reviewed
now and every 6 months, for up to 24 months.
Yes
No
4. To have a study team member access the child’s service records including records from
Medicaid, Head Start, mental health, and early intervention services. The purpose of
accessing these records is to identify all services received in the 12-months prior to the
study and during the course of the study period and the costs of those services.
Yes
No
Child Welfare Administrator (Type or Print Full Name):
______________________________________
Signature of Child Welfare Administrator:
____________________________________________________

Date: ___/___/____

Name of Child/Youth (Print) ________________________________

Date: ___/___/____