Form Stakeholder Interv Stakeholder Interv Stakeholder Interviews

National System of Care (SOC) Expansion Evaluation

Attachment 2 Semi-Structured Stakeholder Interviews 12-17-14

Stakeholder Interviews

OMB: 0930-0349

Document [docx]

Download: docx | pdf

Attachment 2

OMB No. 0930-03xx

Expiration Date: xx/xx/xx

Shape1

CHILDREN’S MENTAL HEALTH INITIATIVE

NATIONAL SYSTEM OF CARE EXPANSION EVALUATION

semi-structured stakeholder interviews

Public Burden Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this project is 0930-03xx. Public reporting burden for this collection of information is estimated to average 80 minutes per respondent, per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to SAMHSA Reports Clearance Officer, 1 Choke Cherry Road, Room 2-1057, Rockville, Maryland, 20857.

Attachment 2a

CMHI SOC Evaluation: Overview of the Stakeholder Interviews

Stakeholder Interview Purpose

The purpose of the stakeholder interviews is to develop a rich qualitative description of grantee implementation and expansion plans and grant efforts. This evaluation component focuses on jurisdiction-level activities. Of particular interest are the strategies employed to implement and expand system of care across the broad jurisdiction. Factors that impede and facilitate those efforts will also be explored.

Development of Stakeholder Interview Questions Content

The stakeholder interviews were developed from a conceptual framework that included service system components and system of care principles (listed below). Indicators were derived from this framework to capture how each system of care principle is enacted to achieve the goals within each service system component. From this framework, interview questions were developed to collect data to inform those indicators.

The following service system components are noted as critical to understanding system of care implementation and expansion:

Governance: The governing structure responsible for explicating the system’s goals, vision, and mission; strategic planning and policy development; and establishing formal arrangements among public agencies and family organizations. These structures may include boards of directors, oversight/steering committees, and interagency boards and structures.
Management: Policies and procedures, developed at the jurisdiction-level, that impact service delivery across the jurisdiction.
Support of Local Service Delivery: The extent to which jurisdiction-wide policies, procedures, and strategies impact local service delivery.
Geographic Area Covered: The physical area encompassed by the jurisdiction.

The following system of care principles informed the indicators in the framework and the questions in the interview protocol:

Family-Driven: The recognition that (a) the ecological context of the family is central to the care of all children; (b) families are primary decision makers and equal partners in all efforts to serve children; and (c) all system and service processes should be planned to maximize family involvement and decision making. Also included is the importance of meeting the needs of families as they care for their children through formal and informal services and supports.
Youth-Guided: The recognition that young people have a right to be empowered, educated, and given the opportunity to make decisions about their own care, and about the policies and procedures governing the care of all youth.
Individualized: The provision of care that is expressly individualized and person-centered, that addresses the child or youth’s specific needs, and that recognizes and incorporates the child or youth’s strengths.
Evidence-Supported: The provision of services that include evidence-informed and promising practices, as well as interventions supported by practice-based evidence, to ensure the effectiveness of services and improve outcomes for children and their families
Culturally and Linguistically Competent: Sensitivity and responsiveness to, and acknowledgment of, the inherent value of differences related to race, ethnicity, religion, language, national origin, gender identification, sexual orientation, socio-economic background, and certain community-specific and family-specific characteristics. Recognition that providing culturally competent care requires the involvement of diverse perspectives in decision-making related to policies and procedures.
Interagency/Collaborative: The involvement and partnership of core agencies from multiple child-serving sectors, including child welfare, health, juvenile justice, intellectual disabilities, substance abuse, education, and mental health.
Coordination: Professionals working together in a complimentary manner to avoid duplication of services, eliminate gaps in care, and facilitate the child, youth and family’s movement through the service system. This includes non-governmental agencies such as private providers, community organizations, churches, advocacy groups, etc.
Accessible: The minimizing of barriers to services in terms of physical location, convenience of scheduling, financial constraints, and perceptions (e.g., stigma).
Community Based: The provision of services and supports within close geographical proximity to the intended population.
Least Restrictive: The priority that services should be delivered in settings that maximize freedom of choice and movement, and interaction in normative environments such as school and family.

We also examined the following components more broadly:

Quality Monitoring: Quality management throughout the system conducted through formal collection and analysis of process and outcome data, and the use of continuous feedback loops to improve jurisdiction-level system performance.
Barriers and Facilitators: Factors that impede and facilitate implementing system of care principles and service components.

Stakeholder Interview Respondents

Respondents for the stakeholder interviews will be top administrators involved in the expansion grant. Interviews will be conducted with:

The CMHI Project Director
The Family Representative
The Youth Representative
The Mental Health Agency Representative
The Child Welfare Agency Representative
The Juvenile Justice Agency Representative
The Education Agency Representative
The Individual Responsible for CMHI Quality Improvement

Method of Data Collection

Interviews will be conducted with single respondents (no group interviews) by phone, Skype, and/or video conference in the final quarter of the first 12 months of the funding (Time 1) and in the last 12-18 months of the grant (Time 2). The interview questions are exactly the same for both protocols at Time 1 and Time 2, except for the 4-5 questions in the Introduction section at the beginning of each protocol. The differences between the Time 1 and Time 2 protocols are as follows. In the Time 1 protocol, the questions in the Introduction section ask respondents about (1) what they hope to accomplish with the grant, (2) the current goals of the grant, (3) their planned activities to accomplish those goals, (4) their grant structure and team members, and (5) how they define expansion. The questions in the Introduction section of the Time 2 protocol are slightly different in that they ask for a description of grantee progress. Specifically, the questions ask grantees (1) what they accomplished during the grant, (2) what their goals were, (3) what their progress was in achieving those goals, and (4) what their progress was in achieving expansion.

Attachment 2b

Shape2

INFORMED CONSENT

These interviews will be conducted with grantee representatives over the phone. Thus, information about the purpose and scope of the evaluation will be shared at the beginning of the interview over the phone. The respondent will be asked if he/she understands the information shared and voluntarily agrees to respond to the questions. Thus, respondents will provide verbal consent.

Attachment 2c

OMB No. xxxx-xxxx

Expiration Date: xx/xx/201x

Shape3

CHILDREN’S MENTAL HEALTH INITIATIVE

NATIONAL SYSTEM OF CARE EXPANSION EVALUATION

Semi-structured stakeholder interviews

project director Version

INTRODUCTION

Thank you for your willingness to participate in the Stakeholder Interview. The purpose of this interview is to develop a rich qualitative description of grantee implementation and expansion plans and early efforts from a project director’s perspective.

Shape4

CONFIDENTIALITY/INFORMED CONSENT

The National Evaluation team is conducting an evaluation of system of care expansion grantees.

We will be asking you to share your opinions and ideas about various topics related to system of care implementation and expansion.

This session will last approximately 1 hour and 30 minutes.

Your participation is completely voluntary, and you have the right to stop at any time or to refuse to answer any question.

Your responses to these questions will be kept confidential and will not be shared outside of the evaluation team. In any of our reports, your responses will be combined with other people’s responses, so your answers will never be attributed to your name.

[OBTAIN INFORMED CONSENT]

We would like to record this interview so that we can be sure to accurately capture your responses. A recording would only be reviewed by a small number of National Evaluation staff members.

[OBTAIN CONSENT TO RECORD SESSION]

INSTRUCTIONS

This Stakeholder Interview focuses on jurisdiction-level activities. By “jurisdiction” we mean the broad state, multi-county, territory, or tribal area that received federal grant funds within which smaller community areas are providing direct services to children, youth and families.

In your system of care, the term jurisdiction refers to [insert tailored definition of jurisdiction for grantee respondent].

As I ask you the interview questions, please ask for any clarification as needed and provide as accurate information as possible.

OMB No. xxxx-xxxx

Expiration Date: xx/xx/201x

Introduction Questions for Time 1 Administration (during the first 12 months of the grant):

Before we get started, could you please describe your position and role as it relates to the [name of system of care expansion grant]? Could you please tell me how long you have been in this position?

Overall what do you hope to accomplish or achieve with your grant?

Please describe what your goals are for the grant.

Could you tell us what activities you intend to do to accomplish those goals?

How is your grant structured and who are the main people or groups involved?

How do you define “expansion” in the context of your grant effort? Specifically, what areas does the grant intend to expand (e.g., geographical area, additional populations, additional sectors/agencies, innovative funding mechanisms, additional evidence-based treatments, etc.)?

Introduction Questions for Time 2 Administration (during the last 12-18 months of the grant):

Before we get started, could you please describe your position and role as it relates to the [name of system of care expansion grant]? Could you please tell me how long you have been in this position?

Please describe what you accomplished or achieved with your grant.
Please describe what your goals were for the grant.
Please describe your progress in achieving those goals. What activities have contributed to this progress?
Please describe the progress of your expansion efforts.

The remaining questions are asked at BOTH Time 1 and Time 2

Subsection 1: Governance

The following questions are about system of care governance. By governance we mean, the jurisdiction-level governing body/structure responsible for 1) developing the system of care’s goals, vision, mission, 2) establishing strategic planning and policy development; and 3) establishing formal arrangements among public agencies, partner organizations, including youth and family organizations. These structures may include boards of directors, oversight/steering committees, and interagency boards and structures. [INTERVIEWER: Provide a definition of system of care governance specific to the grantee being interviewed].

Subsection 1: Governance

1. How are family members/family organizations actively involved as partners in system of care governance at the jurisdiction-level?

Is there family member and/or family organization representation on the system of care implementation team? Do they have voting power? Do they regularly attend system of care planning and implementation meetings?

What committees and workgroups do family members participate in (for example, social marketing or evaluation workgroups, or decision making group for budgets)? What other roles do they serve (for example, leadership positions)?

2. What jurisdiction-level strategies are in place to support family involvement in system of care governance?

Probe for: Policies requiring family participation (for example, governing body bylaws, other rules); orientation and training and/or mentoring; compensation for time; assistance with child care; compensation for travel; consideration of family members’ schedules when scheduling meeting times; other supports

3. How are youth/youth organizations actively involved as partners in the system of care governance at the jurisdiction-level?

Is there youth member and/or youth organization representation on the system of care governance body? Do they have voting power on the governance body? Do they regularly attend system of care planning and implementation meetings?

What committees and workgroups do youth members participate in (for example, social marketing or evaluation workgroups, or decision making group for budgets)? What other roles do they serve (for example, leadership positions)?

4. What jurisdiction-level strategies are in place for supporting youth involvement in system of care governance?

Probe for: Policies regarding youth participation (for example, governing body bylaws, other rules); orientation and training and/or mentoring; compensation for time; assistance with child care; compensation for travel; consideration of youth’s schedules when scheduling meeting times; other supports

5. What jurisdiction-level strategies are in place to ensure that system of care governance is culturally and linguistically competent?

Probe for: Promotion of cultural, linguistic, and ethnic diversity on governing bodies; accommodations for language differences in meetings and communications; members representing population of focus; training

Subsection 1: Governance (Continued)

6. What child- and family-serving agencies are involved in system of care governance at the jurisdiction level (for example, mental health, child welfare, juvenile justice, intellectual disabilities, education, others)?

How are representatives from child- and family-serving agencies involved in governance?

Is their involvement in the governance structure supported by MOUs or other formal agreements?

7. What jurisdiction-level strategies are in place to support interagency and organizational collaboration in governance?

Subsection 2: Management, Policies, Procedures

The following questions are about management, policies and procedures developed at the jurisdiction-level that impact operations and service delivery across the jurisdiction.

8. Does having “lived experience” in raising a child with mental health challenges play a role in hiring decisions for staff?

How would you learn about that type of “lived experience” from job applicants?

9. In what ways are family/family organizations involved in management decision making?

Probe for: Family member/family organization representation on the management team, and in what roles, and what level of decision making power?

How does the SOC support that involvement?

10. Does having “lived experience” as a youth who has had mental health challenges play a role when making hiring decisions for staff?

How would you learn about that type of “lived experience” from job applicants?

11. In what ways are youth/youth organizations involved in management decision making?

Probe for: Youth member/youth organization representation on the jurisdiction-level management team, and in what roles, and what level of decision making power?

How does the SOC support that involvement?

Subsection 2: Management, Policies, Procedures (Continued)

12. What efforts are made to promote cultural, linguistic and ethnic diversity in jurisdiction-level management, policies, and procedures?

To what extent is jurisdiction-level management staff culturally and ethnically diverse?

In what way does the cultural and ethnic profile of the jurisdiction-level management staff reflect the makeup of the population of children, youth and families served? What are the gaps? What plans if any are there to address gaps?

13. What jurisdiction-level strategies are underway for expanding service accessibility across the jurisdiction?

Probe for: Outreach efforts; jurisdiction-wide policies regarding times and locations of services, transportation, colocation, and telemedicine; insurance enrollment navigators, financial supports, social marketing

14. What jurisdiction-level strategies are in place to ensure that children and families receive clinical and support services in their home communities?

15. What measures have been taken at the jurisdiction level to reduce the use of residential services?

Examples include having intermediate services available (for example, day treatment, treatment foster care); requiring a discharge plan at the time of admission that includes target discharge date; having a utilization review process; having case review teams for children at high risk for residential placement

Subsection 3: Support of Local Service Delivery

The following questions are about the extent to which jurisdiction-wide policies, procedures, and strategies impact service delivery at the local system level.

16. What jurisdiction-level policies or strategies are in place to support/promote the use of family members as local service system staff?

What roles do family members play in the local service system (for example, administrative, service providers, peer support)?

17. What jurisdiction-level strategies are in place to ensure that a full range of family services and supports are available in the service array? What types of family services and supports are being provided?

Examples of “family services and supports” include peer-parent support, respite care

How are these services and supports funded?

Subsection 3: Support of Local Service Delivery (Continued)

18. What jurisdiction-level policies or strategies are in place to support/promote the use of youth as local service system staff?

What roles do youth play in the local service system (for example, administrative, service providers, peer support)?

19. What jurisdiction-level strategies are in place to ensure that a full range of youth services and supports are available in the service array? What types of youth services and supports are being provided?

Examples of “youth services and supports” include youth peer-to-peer services, youth groups, youth advocates, mentoring

How are these services and supports funded?

20. What jurisdiction-level policies or strategies are in place to ensure individualized care in service delivery within local systems?

Examples of individualized care policies include requiring wraparound or other person-centered service planning models, contracting with care management entities, providing trainings on individualized care, etc.

21. What jurisdiction-level strategies are in place to support the delivery of evidenced-supported practices and supports at the local service delivery level?

Examples of “evidence-supported services and supports” include, but are not limited to, manualized peer to peer support models or other family support models; parent-child interaction therapy (PCIT); functional family therapy (FFT); parent training; behavior management training, etc.

Probe for: Adoption of evidence-based service/treatment models across jurisdiction; jurisdiction-level training and implementation support for evidence-based practices; formation of task forces focused on evidence-based practice implementation

22. How does the jurisdiction support local system outreach to underserved populations?

23. How does the jurisdiction support local service systems in the delivery of services in a culturally and linguistically competent manner?

Probe for: Training of providers and staff; jurisdiction-wide hiring policies; addition of service options to the jurisdiction-wide service array to meet the needs of specific cultural groups

Subsection 3: Support of Local Service Delivery (Continued)

24. What jurisdiction-level strategies are in place to support interagency and organizational collaboration in local service systems?

Probe for: Financing arrangements; state-level policies; MOUs; state policies that promote co-location of staff, out-stationing of staff to other agency offices

25. What jurisdiction-level strategies are in place to support effective care coordination in local service systems?

Examples include mandating the use of wraparound across the state; establishing maximum or minimum caseload sizes; including a Medicaid billing code for multiple individuals in case meetings

26. How does the jurisdiction support local systems in their efforts to improve accessibility of services?

Subsection 4: Geographic Area Covered

27. How does the jurisdiction support local service systems in their efforts to serve children in their home communities?

The following questions are about the geographic area encompassed by the jurisdiction for your system of care expansion grant.

28. What jurisdiction-level strategies are in place to ensure that families or family organizations are involved in planning, implementation and expansion efforts across the jurisdiction (as opposed to select areas)?

To what extent are families active in system work across the whole jurisdiction (for example, state, territory, tribal area, county)?

29. What jurisdiction-level strategies are in place to ensure that youth or youth organizations are involved in planning, implementation, and expansion efforts across the jurisdiction (as opposed to select areas)?

To what extent are youth active in system work across the whole jurisdiction (for example, state, territory, tribal area, county)?

30. What jurisdiction-level activities are occurring to support multiagency system of care efforts across the jurisdiction?

Are there areas in the jurisdiction where interagency collaboration is greater or less active? If yes, describe.

Subsection 5: Continuous Quality Improvement

The following questions are about quality improvement activities, including the formal collection, analysis of process and outcome data, and the use of that information to improve jurisdiction-level system performance.

31. What data systems or quality monitoring strategies are in place in the system of care? Is it a routine or ongoing process? What areas were assessed in the past year?

32. How is quality monitoring information used to support system performance at the jurisdiction level?

Probe for: Who utilizes the information (for example, organizations or agencies).

Subsection 6: Barriers/Facilitators

The next set of questions is about facilitators and barriers to your system of care implementation, planning and expansion efforts.

33. What factors have facilitated your system of care efforts at the jurisdiction level?

34. What barriers have you encountered in your system of care efforts at the jurisdiction level?

Shape5

End of Instrument

Thank you for participating in the stakeholder interview portion of the National Systems of Care Expansion Evaluation.

Attachment 2d

OMB No. xxxx-xxxx

Expiration Date: xx/xx/201x

Shape6

CHILDREN’S MENTAL HEALTH INITIATIVE

NATIONAL SYSTEM OF CARE EXPANSION EVALUATION

Semi-structured stakeholder interviews

Mental health director Version

INTRODUCTION

Shape7

CONFIDENTIALITY/INFORMED CONSENT

The National Evaluation team is conducting an evaluation of system of care expansion grantees.

We will be asking you to share your opinions and ideas about various topics related to system of care implementation and expansion.

This session will last approximately 60 minutes.

Your participation is completely voluntary, and you have the right to stop at any time or to refuse to answer any question.

[OBTAIN INFORMED CONSENT]

We would like to record this interview so that we can be sure to accurately capture your responses. A recording would only be reviewed by a small number of National Evaluation staff members.

[OBTAIN CONSENT TO RECORD SESSION]

INSTRUCTIONS

This Stakeholder Interview focuses on jurisdiction-level activities. By “jurisdiction” we mean the broad state, multi-county, territory, or tribal area that received federal grant funds within which smaller community areas are providing direct services to children, youth and families.

In your system of care, the term jurisdiction refers to [insert tailored definition of jurisdiction for grantee respondent].

As I ask you the interview questions, please ask for any clarification as needed and provide as accurate information as possible.

OMB No. xxxx-xxxx

Expiration Date: xx/xx/201x

Introduction Questions for Time 1 Administration (during the first 12 months of the grant):

Before we get started, could you please describe your position and role as it relates to the [name of system of care expansion grant]? Could you please tell me how long you have been in this position?

Overall what do you hope to accomplish or achieve with your grant?

Please describe what your goals are for the grant.

Could you tell us what activities you intend to do to accomplish those goals?

How is your grant structured and who are the main people or groups involved?

How do you define “expansion” in the context of your grant effort? Specifically, what areas does the grant intend to expand (e.g., geographical area, additional populations, additional sectors/agencies, innovative funding mechanisms, additional evidence-based treatments, etc.)?

Introduction Questions for Time 2 Administration (during the last 12-18 months of the grant):

Before we get started, could you please describe your position and role as it relates to the [name of system of care expansion grant]? Could you please tell me how long you have been in this position?

Please describe what you accomplished or achieved with your grant.
Please describe what your goals were for the grant.
Please describe your progress in achieving those goals. What activities have contributed to this progress?
Please describe the progress of your expansion efforts.

The remaining questions are asked at BOTH Time 1 and Time 2

Subsection 1: Governance

Subsection 1: Governance (Continued)

1. How are family members/family organizations actively involved as partners in system of care governance at the jurisdiction level?

2. How are youth/youth organizations actively involved as partners in system of care governance at the jurisdiction level?

3. What jurisdiction-level strategies are in place to ensure that system of care governance is culturally and linguistically competent?

4. What child- and family-serving agencies are involved in system of care governance at the jurisdiction level (for example, mental health, child welfare, juvenile justice, intellectual disabilities, education, others)?

How are representatives from child- and family-serving agencies involved in governance?

Is their involvement in the governance structure supported by MOUs or other formal agreements?

5. What jurisdiction-level strategies are in place to support interagency and organizational collaboration in governance?

Subsection 2: Management, Policies, Procedures

The following questions are about management, policies and procedures developed at the jurisdiction-level that impact operations and service delivery across the jurisdiction.

6. What efforts are made at the jurisdiction level to promote cultural, linguistic and ethnic diversity in jurisdiction-level management, policies, and procedures?

To what extent is jurisdiction-level management staff culturally and ethnically diverse?

In what way does the cultural and ethnic profile of the jurisdiction level management staff reflect the makeup of the population of children, youth and families served? What are the gaps? What plans if any are there to address gaps?

7. What jurisdiction-level strategies are underway for expanding service accessibility across the jurisdiction?

8. What jurisdiction-level strategies are in place to ensure that children and families receive clinical and support services in their home communities?

9. What measures have been taken at the jurisdiction level to reduce the use of residential services?

Subsection 3: Support of Local Service Delivery

The following questions are about the extent to which jurisdiction-wide policies, procedures, and strategies impact service delivery at the local system level.

10. What jurisdiction-level policies or strategies are in place to support/promote the use of family members as local service system staff?

What roles do family members play in the local service system (for example, administrative, service providers, peer support)?

Subsection 3: Support of Local Service Delivery (Continued)

11. What jurisdiction-level strategies are in place to ensure that a full range of family services and supports are available in the service array? What types of family services and supports are being provided?

Examples of “family services and supports” include peer-parent support, respite care

How are these services and supports funded?

12. What jurisdiction-level policies or strategies are in place to support/promote the use of youth as local service system staff?

What roles do youth play in the local service system (for example, administrative, service providers, peer support)?

13. What jurisdiction-level strategies are in place to ensure that a full range of youth services and supports are available in the service array? What types of youth services and supports are being provided?

Examples of “youth services and supports” include youth peer-to-peer services, youth groups, youth advocates, mentoring

How are these services and supports funded?

14. What jurisdiction-level policies or strategies are in place to ensure individualized care in service delivery within local systems?

15. What jurisdiction-level strategies are in place to support the delivery of evidenced-supported practices and supports at the local service delivery level?

16. How does the jurisdiction support local system outreach to underserved populations?

Subsection 3: Support of Local Service Delivery (Continued)

17. How does the jurisdiction support local service systems in the delivery of services in a culturally and linguistically competent manner?

Probe for: Training of providers and staff; jurisdiction-wide hiring policies; addition of service options to the jurisdiction-wide service array to meet the needs of specific cultural groups

18. What jurisdiction-level strategies are in place to support interagency and organizational collaboration in local service systems?

Probe for: Financing arrangements; state-level policies; MOUs; state policies that promote co-location of staff, out stationing of staff to other agency offices

19. What jurisdiction-level strategies are in place to support effective care coordination in local service systems?

Examples include mandating the use of wraparound across the state; establishing maximum or minimum caseload sizes; including a Medicaid billing code for multiple individuals in case meetings

20. How does the jurisdiction support local systems in their efforts to improve accessibility of services?

21. How does the jurisdiction support local service systems in their efforts to serve children in their home communities?

Subsection 4: Geographic Area Covered

The following questions are about the geographic area encompassed by the jurisdiction for your system of care expansion grant.

22. What jurisdiction-level strategies are in place to ensure that families or family organizations are involved in planning, implementation and expansion efforts across the jurisdiction (as opposed to select areas)?

To what extent are families active in system work across the whole jurisdiction (for example, state, territory, tribal area, county)?

23. What activities are occurring at the jurisdiction level to support multiagency system of care efforts across the jurisdiction?

Are there areas in the jurisdiction where interagency collaboration is greater or less active? If yes, describe.

Subsection 5: Continuous Quality Improvement

24. How is quality monitoring information used to support system performance at the jurisdiction level?

Probe for: Who utilizes the information (for example, organizations or agencies).

Subsection 6: Barriers/Facilitators

The next set of questions is about facilitators and barriers to your system of care implementation, planning and expansion efforts.

25. What factors have facilitated your system of care efforts at the jurisdiction level?

26. What barriers have you encountered in your system of care efforts at the jurisdiction level?

End of Instrument:

Thank you for participating in the stakeholder interview portion of the National Systems of Care Expansion Evaluation.

Attachment 2e

OMB No. xxxx-xxxx

Expiration Date: xx/xx/201x

Shape9

CHILDREN’S MENTAL HEALTH INITIATIVE

NATIONAL SYSTEM OF CARE EXPANSION EVALUATION

Semi-structured stakeholder interviews

Education and Juvenile justice Respondent Version

INTRODUCTION

Shape10

CONFIDENTIALITY/INFORMED CONSENT

The National Evaluation team is conducting an evaluation of system of care expansion grantees.

We will be asking you to share your opinions and ideas about various topics related to system of care implementation and expansion.

This session will last approximately 60 minutes.

Your participation is completely voluntary, and you have the right to stop at any time or to refuse to answer any question.

[OBTAIN INFORMED CONSENT]

We would like to record this interview so that we can be sure to accurately capture your responses. A recording would only be reviewed by a small number of National Evaluation staff members.

[OBTAIN CONSENT TO RECORD SESSION]

INSTRUCTIONS

In your system of care, the term jurisdiction refers to [insert tailored definition of jurisdiction for grantee respondent].

As I ask you the interview questions, please ask for any clarification as needed and provide as accurate information as possible.

OMB No. xxxx-xxxx

Expiration Date: xx/xx/201x

Introduction Questions for Time 1 Administration (during the first 12 months of the grant):

Before we get started, could you please describe your position and role as it relates to the [name of system of care expansion grant]? Could you please tell me how long you have been in this position?

Overall what do you hope to accomplish or achieve with your grant?

Please describe what your goals are for the grant.

Could you tell us what activities you intend to do to accomplish those goals?

How is your grant structured and who are the main people or groups involved?

How do you define “expansion” in the context of your grant effort? Specifically, what areas does the grant intend to expand (e.g., geographical area, additional populations, additional sectors/agencies, innovative funding mechanisms, additional evidence-based treatments, etc.)?

Introduction Questions for Time 2 Administration (during the last 12-18 months of the grant):

Before we get started, could you please describe your position and role as it relates to the [name of system of care expansion grant]? Could you please tell me how long you have been in this position?

Please describe what you accomplished or achieved with your grant.
Please describe what your goals were for the grant.
Please describe your progress in achieving those goals. What activities have contributed to this progress?
Please describe the progress of your expansion efforts.

The remaining questions are asked at BOTH Time 1 and Time 2

Subsection 1: Governance

Subsection 1: Governance (Continued)

1. How are family members/family organizations actively involved as partners in system of care governance at the jurisdiction level?

2. How are youth/youth organizations actively involved as partners in system of care governance at the jurisdiction level?

3. What child- and family-serving agencies are involved in system of care governance at the jurisdiction level (for example, mental health, child welfare, juvenile justice, education, intellectual disabilities, others)?

How are representatives from child- and family-serving agencies involved in governance?

Is their involvement in the governance structure supported by MOUs or other formal agreements?

4. What jurisdiction-level strategies are in place to support interagency and organizational collaboration in governance?

Subsection 2: Management, Policies, Procedures

The following questions are about management, policies and procedures developed at the jurisdiction-level that impact operations and service delivery across the jurisdiction.

5. What jurisdiction-level strategies are underway for expanding service accessibility across the jurisdiction?

Subsection 2: Management, Policies, Procedures (Continued)

6. What measures have been taken at the jurisdiction level to reduce the use of residential services?

Examples include having intermediate services available (e.g., day treatment, treatment foster care); requiring a discharge plan at the time of admission that includes target discharge date; having a utilization review process; having case review teams for children at high risk for residential placement

Subsection 3: Support of Local Service Delivery

The following questions are about the extent to which jurisdiction-wide policies, procedures, and strategies impact service delivery at the local system level.

7. What jurisdiction-level strategies are in place to ensure that a full range of family services and supports are available in the service array? What types of family services and supports are being provided?

Examples of “family services and supports” include peer-parent support, respite care

How are these services and supports funded?

8. What jurisdiction-level strategies are in place to support the delivery of evidenced-supported practices and supports at the local service delivery level?

9. What jurisdiction-level strategies are in place to support interagency and organizational collaboration in local service systems?

Probe for: Financing arrangements; state-level policies; MOUs; state policies that promote co-location of staff, out stationing of staff to other agency offices

Subsection 3: Support of Local Service Delivery (Continued)

10. What jurisdiction-level strategies are in place to support effective care coordination in local service systems?

Examples include mandating the use of wraparound across the state; establishing maximum or minimum caseload sizes; including a Medicaid billing code for multiple individuals in case meetings

11. How does the jurisdiction support local systems in their efforts to improve accessibility of services?

Subsection 4: Geographic Area Covered

The following questions are about the geographic area encompassed by the jurisdiction for your system of care expansion grant.

12. What jurisdiction-level strategies are in place to ensure that families or family organizations are involved in planning, implementation and expansion efforts across the jurisdiction (as opposed to select areas)?

To what extent are families active in system work across the whole jurisdiction (e.g., state, territory, tribal area, county)?

13. What activities are occurring at the jurisdiction level to support multiagency system of care efforts across the jurisdiction?

Are there areas in the jurisdiction where interagency collaboration is greater or less active? If yes, describe.

Subsection 5: Continuous Quality Improvement

14. How is quality monitoring information used to support system performance at the jurisdiction level?

Probe for: Who utilizes the information (for example, organizations or agencies).

Subsection 6: Barriers/Facilitators

The next set of questions is about facilitators and barriers to your system of care implementation, planning and expansion efforts.

15. What factors have facilitated your system of care efforts at the jurisdiction level?

16. What barriers have you encountered in your system of care efforts at the jurisdiction level?

End of Instrument:

Thank you for participating in the stakeholder interview portion of the National Systems of Care Expansion Evaluation.

Attachment 2f

OMB No. xxxx-xxxx

Expiration Date: xx/xx/201x

Shape12

CHILDREN’S MENTAL HEALTH INITIATIVE

NATIONAL SYSTEM OF CARE EXPANSION EVALUATION

Semi-structured stakeholder interviews

Child Welfare department Version

INTRODUCTION

Shape13

CONFIDENTIALITY/INFORMED CONSENT

The National Evaluation team is conducting an evaluation of system of care expansion grantees.

We will be asking you to share your opinions and ideas about various topics related to system of care implementation and expansion.

This session will last approximately 60 minutes.

Your participation is completely voluntary, and you have the right to stop at any time or to refuse to answer any question.

[OBTAIN INFORMED CONSENT]

We would like to record this interview so that we can be sure to accurately capture your responses. A recording would only be reviewed by a small number of National Evaluation staff members.

[OBTAIN CONSENT TO RECORD SESSION]

INSTRUCTIONS

In your system of care, the term jurisdiction refers to [insert tailored definition of jurisdiction for grantee respondent].

As I ask you the interview questions, please ask for any clarification as needed and provide as accurate information as possible.

OMB No. xxxx-xxxx

Expiration Date: xx/xx/201x

Introduction Questions for Time 1 Administration (during the first 12 months of the grant):

Before we get started, could you please describe your position and role as it relates to the [name of system of care expansion grant]? Could you please tell me how long you have been in this position?

Overall what do you hope to accomplish or achieve with your grant?

Please describe what your goals are for the grant.

Could you tell us what activities you intend to do to accomplish those goals?

How is your grant structured and who are the main people or groups involved?

How do you define “expansion” in the context of your grant effort? Specifically, what areas does the grant intend to expand (e.g., geographical area, additional populations, additional sectors/agencies, innovative funding mechanisms, additional evidence-based treatments, etc.)?

Introduction Questions for Time 2 Administration (during the last 12-18 months of the grant):

Before we get started, could you please describe your position and role as it relates to the [name of system of care expansion grant]? Could you please tell me how long you have been in this position?

Please describe what you accomplished or achieved with your grant.
Please describe what your goals were for the grant.
Please describe your progress in achieving those goals. What activities have contributed to this progress?
Please describe the progress of your expansion efforts.

The remaining questions are asked at BOTH Time 1 and Time 2

Subsection 1: Governance

Subsection 1: Governance (Continued)

1. How are family members/family organizations actively involved as partners in system of care governance at the jurisdiction level?

2. How are youth/youth organizations actively involved as partners in system of care governance at the jurisdiction level?

How are representatives from child- and family-serving agencies involved in governance?

Is their involvement in the governance structure supported by MOUs or other formal agreements?

4. What jurisdiction-level strategies are in place to support interagency and organizational collaboration in governance?

Subsection 2: Management, Policies, Procedures

The following questions are about management, policies and procedures developed at the jurisdiction-level that impact operations and service delivery across the jurisdiction.

5. What jurisdiction-level strategies are underway for expanding service accessibility across the jurisdiction?

Subsection 2: Management, Policies, Procedures (Continued)

6. What jurisdiction-level strategies are in place to ensure that children and families receive clinical and support services in their home communities?

7. What measures have been taken at the jurisdiction level to reduce the use of residential services?

Subsection 3: Support of Local Service Delivery

The following questions are about the extent to which jurisdiction-wide policies, procedures, and strategies impact service delivery at the local system level.

8. What jurisdiction-level strategies are in place to ensure that a full range of family services and supports are available in the service array? What types of family services and supports are being provided?

Examples of “family services and supports” include peer-parent support, respite care

How are these services and supports funded?

9. What jurisdiction-level strategies are in place to support the delivery of evidenced-supported practices and supports at the local service delivery level?

10. What jurisdiction-level strategies are in place to support interagency and organizational collaboration in local service systems?

Probe for: Financing arrangements; state-level policies; MOUs; state policies that promote co-location of staff, out stationing of staff to other agency offices

Subsection 3: Support of Local Service Delivery (Continued)

11. What jurisdiction-level strategies are in place to support effective care coordination in local service systems?

Examples include mandating the use of wraparound across the state; establishing maximum or minimum caseload sizes; including a Medicaid billing code for multiple individuals in case meetings

12. How does the jurisdiction support local systems in their efforts to improve accessibility of services?

13. How does the jurisdiction support local service systems in their efforts to serve children in their home communities?

Subsection 4: Geographic Area Covered

The following questions are about the geographic area encompassed by the jurisdiction for your system of care expansion grant.

14. What jurisdiction-level strategies are in place to ensure that families or family organizations are involved in planning, implementation and expansion efforts across the jurisdiction (as opposed to select areas)?

To what extent are families active in system work across the whole jurisdiction (e.g., state, territory, tribal area, county)?

15. What activities are occurring at the jurisdiction level to support multiagency system of care efforts across the jurisdiction?

Are there areas in the jurisdiction where interagency collaboration is greater or less active? If yes, describe.

Subsection 5: Continuous Quality Improvement

16. How is quality monitoring information used to support system performance at the jurisdiction level?

Probe for: Who utilizes the information (for example, organizations or agencies).

Subsection 6: Barriers/Facilitators

The next set of questions is about facilitators and barriers to your system of care implementation, planning and expansion efforts.

17. What factors have facilitated your system of care efforts at the jurisdiction level?

18. What barriers have you encountered in your system of care efforts at the jurisdiction level?

Shape14

End of Instrument:

Thank you for participating in the stakeholder interview portion of the National Systems of Care Expansion Evaluation.

Attachment 2g

OMB No. xxxx-xxxx

Expiration Date: xx/xx/201x

Shape15

CHILDREN’S MENTAL HEALTH INITIATIVE

NATIONAL SYSTEM OF CARE EXPANSION EVALUATION

Semi-structured stakeholder interviews

family representative Version

INTRODUCTION

Shape16

CONFIDENTIALITY/INFORMED CONSENT

The National Evaluation team is conducting an evaluation of system of care expansion grantees.

We will be asking you to share your opinions and ideas about various topics related to system of care implementation and expansion.

This session will last approximately 1 hour and 15 minutes.

Your participation is completely voluntary, and you have the right to stop at any time or to refuse to answer any question.

[OBTAIN INFORMED CONSENT]

We would like to record this interview so that we can be sure to accurately capture your responses. A recording would only be reviewed by a small number of National Evaluation staff members.

[OBTAIN CONSENT TO RECORD SESSION]

INSTRUCTIONS

In your system of care, the term jurisdiction refers to [insert tailored definition of jurisdiction for grantee respondent].

As I ask you the interview questions, please ask for any clarification as needed and provide as accurate information as possible.

OMB No. xxxx-xxxx

Expiration Date: xx/xx/201x

Introduction Questions for Time 1 Administration (during the first 12 months of the grant):

Before we get started, could you please describe your position and role as it relates to the [name of system of care expansion grant]? Could you please tell me how long you have been in this position?

Overall what do you hope to accomplish or achieve with your grant?

Please describe what your goals are for the grant.

Could you tell us what activities you intend to do to accomplish those goals?

How is your grant structured and who are the main people or groups involved?

How do you define “expansion” in the context of your grant effort? Specifically, what areas does the grant intend to expand (e.g., geographical area, additional populations, additional sectors/agencies, innovative funding mechanisms, additional evidence-based treatments, etc.)?

Introduction Questions for Time 2 Administration (during the last 12-18 months of the grant):

Before we get started, could you please describe your position and role as it relates to the [name of system of care expansion grant]? Could you please tell me how long you have been in this position?

Please describe what you accomplished or achieved with your grant.
Please describe what your goals were for the grant.
Please describe your progress in achieving those goals. What activities have contributed to this progress?
Please describe the progress of your expansion efforts.

The remaining questions are asked at BOTH Time 1 and Time 2

Subsection 1: Governance

Subsection 1: Governance (Continued)

1. How are family members/family organizations actively involved as partners in system of care governance at the jurisdiction level?

2. What jurisdiction-level strategies are in place to support family involvement in system of care governance?

Probe for: Policies requiring family participation (e.g., governing body bylaws, other rules); orientation and training and/or mentoring; compensation for time; assistance with

child care; compensation for travel; consideration of family members’ schedules when scheduling meeting times; other supports

3. What jurisdiction-level strategies are in place to ensure that system of care governance is culturally and linguistically competent?

4. What child- and family-serving agencies are involved in system of care governance at the jurisdiction level (for example, mental health, child welfare, juvenile justice, education, intellectual disabilities, others)?

How are representatives from child- and family-serving agencies involved in governance?

Is their involvement in the governance structure supported by MOUs or other formal agreements?

5. What jurisdiction-level strategies are in place to support interagency and organizational collaboration in governance?

Subsection 2: Management, Policies, Procedures

The following questions are about management, policies and procedures developed at the jurisdiction-level that impact operations and service delivery across the jurisdiction.

6. Does having “lived experience” in raising a child with mental health challenges play a role in hiring decisions for staff?

How would you learn about that type of “lived experience” from job applicants?

Subsection 2: Management, Policies, Procedures (Continued)

7. In what ways are family/family organizations involved in management decision making?

Probe for: Family member/family organization representation on the management team, and in what roles, and what level of decision making power?

How does the SOC support that involvement?

8. How are youth involved in the [system of care name]?

9. What efforts are made to promote cultural, linguistic and ethnic diversity in jurisdiction-level management, policies, and procedures?

To what extent is jurisdiction-level management staff culturally and ethnically diverse?

10. What jurisdiction-level strategies are underway for expanding service accessibility across the jurisdiction?

11. What jurisdiction-level strategies are in place to ensure that children and families receive clinical and support services in their home communities?

12. What measures have been taken at the jurisdiction level to reduce the use of residential services?

Subsection 3: Support of Local Service Delivery

The following questions are about the extent to which jurisdiction-wide policies, procedures, and strategies impact service delivery at the local system level.

13. What jurisdiction-level policies or strategies are in place to support/promote the use of family members as local service system staff?

What roles do family members play in the local service system (e.g., administrative, service providers, peer support)?

Subsection 3: Support of Local Service Delivery (Continued)

14. What jurisdiction-level strategies are in place to ensure that a full range of family services and supports are available in the service array? What types of family services and supports are being provided?

Examples of “family services and supports” include peer-parent support, respite care

How are these services and supports funded?

15. What jurisdiction-level policies or strategies are in place to ensure individualized care in service delivery within local systems?

16. What jurisdiction-level strategies are in place to support the delivery of evidenced-supported practices and supports at the local service delivery level?

17. How does the jurisdiction support local system outreach to underserved populations?

18. How does the jurisdiction support local service systems in the delivery of services in a culturally and linguistically competent manner?

Probe for: Training of providers and staff; jurisdiction-wide hiring policies; addition of service options to the jurisdiction-wide service array to meet the needs of specific cultural groups

19. What jurisdiction-level strategies are in place to support interagency and organizational collaboration in local service systems?

Probe for: Financing arrangements; state-level policies; MOUs; state policies that promote co-location of staff, out stationing of staff to other agency offices

20. What jurisdiction-level strategies are in place to support effective care coordination in local service systems?

Examples include mandating the use of wraparound across the state; establishing maximum or minimum caseload sizes; including a Medicaid billing code for multiple individuals in case meetings

Subsection 3: Support of Local Service Delivery (Continued)

21. How does the jurisdiction support local systems in their efforts to improve accessibility of services?

22. How does the jurisdiction support local service systems in their efforts to serve children in their home communities?

Subsection 4: Geographic Area Covered

The following questions are about the geographic area encompassed by the jurisdiction for your system of care expansion grant.

23. What jurisdiction-level strategies are in place to ensure that families or family organizations are involved in planning, implementation and expansion efforts across the jurisdiction (as opposed to select areas)?

To what extent are families active in system work across the whole jurisdiction (e.g., state, territory, tribal area, county)?

24. What activities are occurring to support multiagency system of care efforts across the jurisdiction?

Are there areas in the jurisdiction where interagency collaboration is greater or less active? If yes, describe.

Subsection 5: Continuous Quality Improvement

25. How is quality monitoring information used to support system performance at the jurisdiction level?

Probe for: Who utilizes the information (for example, organizations or agencies).

Subsection 6: Barriers/Facilitators

The next set of questions is about facilitators and barriers to your system of care implementation, planning and expansion efforts.

26. What factors have facilitated your system of care efforts at the jurisdiction level?

27. What barriers have you encountered in your system of care efforts at the jurisdiction level?

End of Instrument:

Thank you for participating in the stakeholder interview portion of the National Systems of Care Expansion Evaluation.

Attachment 2h

OMB No. xxxx-xxxx

Expiration Date: xx/xx/201x

Shape18

CHILDREN’S MENTAL HEALTH INITIATIVE

NATIONAL SYSTEM OF CARE EXPANSION EVALUATION

Semi-structured stakeholder interviews

youth representative Version

INTRODUCTION

Shape19

CONFIDENTIALITY/INFORMED CONSENT

The National Evaluation team is conducting an evaluation of system of care expansion grantees.

We will be asking you to share your opinions and ideas about various topics related to system of care implementation and expansion.

This session will last approximately 1 hour and 15 minutes.

Your participation is completely voluntary, and you have the right to stop at any time or to refuse to answer any question.

[OBTAIN INFORMED CONSENT]

We would like to record this interview so that we can be sure to accurately capture your responses. A recording would only be reviewed by a small number of National Evaluation staff members.

[OBTAIN CONSENT TO RECORD SESSION]

INSTRUCTIONS

In your system of care, the term jurisdiction refers to [insert tailored definition of jurisdiction for grantee respondent].

As I ask you the interview questions, please ask for any clarification as needed and provide as accurate information as possible.

OMB No. xxxx-xxxx

Expiration Date: xx/xx/201x

Introduction Questions for Time 1 Administration (during the first 12 months of the grant):

Before we get started, could you please describe your position and role as it relates to the [name of system of care expansion grant]? Could you please tell me how long you have been in this position?

Overall what do you hope to accomplish or achieve with your grant?

Please describe what your goals are for the grant.

Could you tell us what activities you intend to do to accomplish those goals?

How is your grant structured and who are the main people or groups involved?

How do you define “expansion” in the context of your grant effort? Specifically, what areas does the grant intend to expand (e.g., geographical area, additional populations, additional sectors/agencies, innovative funding mechanisms, additional evidence-based treatments, etc.)?

Introduction Questions for Time 2 Administration (during the last 12-18 months of the grant):

Before we get started, could you please describe your position and role as it relates to the [name of system of care expansion grant]? Could you please tell me how long you have been in this position?

Please describe what you accomplished or achieved with your grant.
Please describe what your goals were for the grant.
Please describe your progress in achieving those goals. What activities have contributed to this progress?
Please describe the progress of your expansion efforts.

The remaining questions are asked at BOTH Time 1 and Time 2

Subsection 1: Governance

Subsection 1: Governance (Continued)

1. How are youth/youth organizations actively involved as partners in system of care governance at the jurisdiction level?

2. What jurisdiction-level strategies are in place for supporting youth involvement in system of care governance?

Probe for: Policies regarding youth participation (e.g., governing body bylaws, other rules); orientation and training and/or mentoring; compensation for time; assistance with child care; compensation for travel; consideration of youth’s schedules when scheduling meeting times; other supports

3. What jurisdiction-level strategies are in place to ensure that system of care governance is culturally and linguistically competent?

How are representatives from child- and family-serving agencies involved in governance?

Is their involvement in the governance structure supported by MOUs or other formal agreements?

5. What jurisdiction-level strategies are in place to support interagency and organizational collaboration in governance?

Subsection 2: Management, Policies, Procedures

The following questions are about management, policies and procedures developed at the jurisdiction-level that impact operations and service delivery across the jurisdiction.

6. Does having “lived experience” as a youth who has had mental health challenges play a role when making hiring decisions for staff?

How would you learn about that type of “lived experience” from job applicants?

Subsection 2: Management, Policies, Procedures (Continued)

7. In what ways are youth/youth organizations involved in management decision making?

Probe for: Youth member/youth organization representation on the jurisdiction-level management team, and in what roles, and what level of decision making power?

How does the SOC support that involvement?

8. How are families involved in the [system of care name]?

9. What efforts are made to promote cultural, linguistic and ethnic diversity in jurisdiction-level management, policies, and procedures?

To what extent is jurisdiction-level management staff culturally and ethnically diverse?

10. What jurisdiction-level strategies are underway for expanding service accessibility across the jurisdiction?

11. What jurisdiction-level strategies are in place to ensure that children and families receive clinical and support services in their home communities?

12. What measures have been taken at the jurisdiction level to reduce the use of residential services?

Subsection 3: Support of Local Service Delivery

The following questions are about the extent to which jurisdiction-wide policies, procedures, and strategies impact service delivery at the local system level.

Subsection 3: Support of Local Service Delivery (Continued)

13. What jurisdiction-level policies or strategies are in place to support/promote the use of youth as local service system staff?

What roles do youth play in the local service system (e.g., administrative, service providers, peer support)?

14. What jurisdiction-level strategies are in place to ensure that a full range of youth services and supports are available in the service array? What types of youth services and supports are being provided?