Form 1 SCIP 2015 Self-Assessment Final

State Court Improvement Program 2015 Annual Self-Assessment Report

The purpose of this report is to create an opportunity to reflect on what you are doing, why you are doing it and if efforts are having the intended results. Questions are designed to solicit candid responses that help you identify what is working well, areas that need improvement and the type of support that would be most helpful. This is intended to be a helpful tool for you and a helpful tool for us to identify how best to use our resources.

The report is made of 7 sections with corresponding questions. Section I allows you to identify two high resource and or high priority projects and discuss them in-depth from a CQI perspective. Section II focuses on current priority areas and driving forces within your state that may be affecting your work. Section III requests a concise accounting of projects/activities in specific topical areas. Section IV focuses on collaborative efforts. Section V centers on CQI needs. Section VI asks you to do a self-assessment of your CIP’s current capacity. Section VII provides a space for you to report on your timeliness and other performance measures.

1. CQI Analyses of Projects

Identify two (2) of your highest priority/highest resource CIP projects that were in some stage of the CQI process in FY 2015. Review and respond to the questions below about these projects. We understand you may be early in the process and may not be able to answer all of these questions. If applicable, indicate where you were in the process when the fiscal year ended and what plans you have for furthering the work.

Project # 1

Briefly describe the project and indicate the approximate date the CIP began working on it.

1. Identify and assess needs. Think about why you decided to focus on this issue. What is the need you were trying to address? What are the outcomes you were hoping to achieve? What evidence (e.g., data) did you have of the need for improvement?

2. Develop theory of change. Do you have a theory about the causes of the problem? What is your "theory of change" (how do you think your activities/interventions will improve the outcomes)?

3. Develop/select solution. How did you select your activities/interventions (e.g., evidence-based, empirically supported, best-practices, etc¹.)

4. Describe the implementation of the project. What did the CIP do to implement the project? What did others (e.g. judges, attorneys) do? Did you do anything to ensure fidelity of the implementation (that is, anything to ensure the program was implemented as it was supposed to be)?

   1. If the project has not yet been implemented, please briefly describe your intentions/plans for implementation.

5. Describe any monitoring/evaluations/assessments of your project and how you intend to apply the findings. How are you monitoring implementation and changes? What data collection tools/methods did you (will you) use to assess effectiveness? What evidence is there that the activities/intervention were effective? What evidence is there that the activities/intervention were implemented with fidelity? Describe how evaluation/assessments were used to inform the project. Does the intervention need to be adjusted, stopped? Does the problem still exist? Was your theory of change supported?

   1. If the project has not yet been evaluated/assessed, please briefly describe your intentions/plans for evaluation/assessment.

6. Is this project a priority for you in 2016? ☐Yes ☐ No

7. Would you like a CQI consult around this project? ☐Yes ☐ No

Project # 2

Briefly describe the project and indicate the approximate date the CIP began working on it.

1. Identify and assess needs. Think about why you decided to focus on this issue. What is the need you were trying to address? What are the outcomes you were hoping to achieve? What evidence (e.g., data) did you have of the need for improvement?

2. Develop theory of change. Do you have a theory about the causes of the problem? What is your "theory of change" (how do you think your activities/interventions will improve the outcomes)?

3. Develop/select solution. How did you select your activities/interventions (e.g., evidence-based, empirically supported, best-practices, etc.).

4. Describe the implementation of the project. What did the CIP do to implement the project? What did others (e.g. judges, attorneys) do? Did you do anything to ensure fidelity of the implementation (that is, anything to ensure the program was implemented as it was supposed to be)?

   1. If the project has not yet been implemented, please briefly describe your intentions/plans for implementation.

5. Describe any monitoring/evaluations/assessments of your project and how you intend to apply the findings. How are you monitoring implementation and changes? What data collection tools/methods did you (will you) use to assess effectiveness? What evidence is there that the activities/intervention were effective? What evidence is there that the activities/intervention were implemented with fidelity? Describe how evaluation/assessments were used to inform the project. Does the intervention need to be adjusted, stopped? Does the problem still exist? Was your theory of change supported?

   1. If the project has not yet been evaluated/assessed, please briefly describe your intentions/plans for evaluation/assessment.

6. Is this project a priority for you in 2016? ☐Yes ☐ No

7. Would you like a CQI consult around this project? ☐Yes ☐ No

2. Trainings, Projects, and Activities

For questions 1-9, provide a concise description of work completed or underway in FY 2015 (October 2014-September 2015) in the below topical subcategories. For question 1, focus on significant training events or initiatives held or developed in FY 2015 and answer the corresponding questions.

For questions 2-9, indicate (yes/no) if you worked on a project or activity in this area. If the answer is yes, that you conducted a project or activity in the area, please complete the table. If the answer is no, skip to the next question. For each project/activity, please provide a brief description, categorize the project by selecting one of the sub-categories available in the drop down box (e.g., for quality hearings, the sub-categories include court observation/assessment, process improvements, specialty/pilot courts, court orders/title IV-E, mediation, appeals, other) and identify the stage of your work by selecting the appropriate state from the drop down box (identifying and assessing needs, developing a theory of change, selecting a solution, implementing your project, or assessing/evaluating your work)².

Questions 2-9 ask you to describe the purpose of the project or activity and how the project or activity will contribute to continuous quality improvement (CQI) in the identified area. Please use the “other” categories to include specific projects that are important to you but do not necessarily fit as part of the CQI process. If you have a project/activity that fits into multiple categories (e.g., youth engagement and well-being), please choose the category you think fits it best and only report the project once.

1. Trainings

Topical Area	Did you hold or develop a training on this topic?	Who was the target audience?	What were the intended training outcomes?	How did you evaluate this training?
Data	☐Yes ☐No
Hearing quality	☐Yes ☐No
Improving timeliness/ permanency	☐Yes ☐No
Quality legal representation	☐Yes ☐No
Engagement & participation of parties	☐Yes ☐No
Well-being	☐Yes ☐No
ICWA	☐Yes ☐No
Sex Trafficking	☐Yes ☐No
Other:	☐Yes ☐No

2. Data Projects. Data projects include any work with administrative data sets (e.g, AFCARS, SACWIS), data dashboards, data reports, fostering court improvement data, case management systems, and data sharing efforts.

Do you have a data project/activity? ☐Yes ☐ No

Project Description	How would you categorize this project?	Work Stage (if applicable)
(add narrative here)
Do you have data reports that you consistently view? ☐ Yes ☐ No If Yes, around which topics? ☐Hearing quality ☐ Timeliness ☐Permanency ☐Well-being ☐Education ☐ Engagement of youth ☐Engagement of Parents ☐Other Engagement ☐ Quality Legal Representation ☐ICWA ☐DCST ☐Runaway Youth ☐Other:______________ ☐Other: ___________________________________

3. Hearing Quality. Hearing quality projects include any efforts you have made to improve the quality of dependency hearings, including court observation/assessment projects, process improvements, specialty/pilot court projects, projects related to court orders or title IV-E determinations, mediation, or appeals.

Do you have a hearing quality project/activity? ☐ Yes ☐ No

Project Description	How would you categorize this project?	Work Stage (if applicable)

4. Improving Timeliness of Hearings or Permanency Outcomes. Timeliness and permanency projects include any activities or projects meant to improve the timeliness of case processing or achievement of timely permanency. This could include general timeliness, focus on continuances or appeals, working on permanency goals other than APPLA, or focus on APPLA and older youth.

Do you have a Timeliness or permanency project/activity? ☐ Yes ☐ No

Project Description	How would you categorize this project?	Work Stage (if applicable)

5. Quality of Legal Representation. Quality of legal representation projects may include any activities/efforts related to improvement of representation for parents, youth, or the agency. This might include assessments or analyzing current practice, implementing new practice models, working with law school clinics, or other activities in this area.

Do you have a quality legal representation project/activity? ☐ Yes ☐No

Project Description	How would you categorize this project?	Work Stage (if applicable)

6. Engagement & Participation of Parties. Engagement and participation of parties includes any efforts centered around youth, parent, foster family, or caregiver engagement, as well as projects related to notice to relatives, limited English proficiency, or other efforts to increase presence and engagement at the hearing.

Do you have an engagement or participation of parties project/activity? ☐ Yes ☐No

Project Description	How would you categorize this project?	Work Stage (if applicable)

7. Well-Being. Well-being projects include any efforts related to improving the well-being of youth. Projects could focus on education, early childhood development, psychotropic medication, LGBTQ youth, trauma, racial disproportionality/disparity, immigration, or other well-being related topics.

Do you have any projects/activities focused on well-being? ☐ Yes ☐No

Project Description	How would you categorize this project?	Work Stage (if applicable)

8. ICWA. ICWA projects could include any efforts to enhance state and tribal collaboration, state and tribal court agreements, data collection and analysis of ICWA compliance, or ICWA notice projects.

Do you have any projects/activities focused on ICWA? ☐ Yes ☐No

Project Description	How would you categorize this project?	Work Stage (if applicable)

9. Preventing Sex Trafficking and Strengthening Families Act (PSTFSA). PSTFSA projects could include any work around domestic child sex trafficking, the reasonable and prudent parent standard, a focus on runaway youth, focus on normalcy, collaboration with other agencies around this topic, data collection and analysis, data sharing, or other efforts to fully implement the act into practice.

Do you have any projects/activities focused on PSTSFA? ☐ Yes ☐No

Project Description	How would you categorize this project?	Work Stage (if applicable)

3. Priority Areas & CIP Resources

1. What would you consider your top two priority areas for FY 2016?

☐ Data projects ☐ Hearing quality

☐ Timeliness/permanency ☐ Quality of legal representation

☐ Engagement of Parties ☐ Well-being

☐ Preventing Sex Trafficking & Strengthening Families

☐ ICWA ☐ Other:_____________________________

2. Are there any outside driving forces that determine your priorities or consume a lot of your time? (For example, legislative involvement or directives, budget concerns, consent decrees and class action litigation, highly publicized child fatalities, unaccompanied minors, etc.)

4. CIP Collaboration and Participation in Child Welfare Program Planning and Improvement Efforts

10. For FY2014, you described how the CIP planned to assist with and participate in round three of the CFSR and program improvement process. We are interested in your progress or any changes to this plan.

1. Has your plan changed? If so, how?

2. How have you moved this plan forward in FY2015?

3. What barriers have you encountered (if any) in increasing your participation with round three of CFSR?

4. Have you received any technical assistance on this issue? If so, what was it and how was it helpful to you?

11. For FY2014 you described how the CIP will assist with and participate in the CFSP/APSR processes with the child welfare agency in an ongoing fashion. We are interested in your progress or any changes to this plan.

1. Has your plan changed? If so, how?

2. How have you moved this plan forward in FY2015?

3. What barriers have you encountered (if any) to working with the child welfare agency in the CFSP/APSR process in an ongoing fashion?

4. Have you received any technical assistance on this issue? If so, what was it and how was it helpful to you?

12. How are you involved, if at all, with the child welfare agency’s CQI efforts?

☐ Contributing data ☐Receiving data ☐Jointly using data

☐ Collaborative meetings ☐ Collaborative systems change project(s)

☐ Other:__________________________________

5. CQI Current Capacity Assessment

1. How is the CIP progressing with CQI overall? Please provide a brief description of how you integrate CQI into your work.

2. Do you have any of the following resources to help you integrate CQI into practice?

☐CIP staff with CQI (e.g., data, evaluation) expertise

☐Consultants with CQI expertise ☐a University partnership

☐Contracts with external agencies to assist with CQI efforts

☐Other resources:_________________________________________

3. Describe the largest challenges your CIP faces with implementing CQI into your work.

4. Please review the list of capacities below. Select the three capacity areas that you would like to increase your knowledge of or enhance your ability to do in the next fiscal year.

☐CQI generally ☐Data collection methodologies ☐ Data analysis ☐Understanding/applying data

☐ Evaluation design ☐Tool development ☐Policy change implementation ☐CQI commitment (buy-in) ☐Collaboration w/agencies ☐Data-driven decision-making

☐Participation in CFSR process ☐Performance measurement

☐Participation in CFSP/APSR process ☐Community partnerships

☐Awareness of evidence-based practices ☐Research partnerships

☐Leadership ☐Data systems

☐Currently available data (e.g., AFCARS) ☐Tracking implementation/changes

☐Training evaluation

Evaluation/CQI efforts specific to:

☐Preventing Trafficking and Strengthening Families Act

☐Quality legal representation ☐Hearing quality

☐Timeliness/Permanency ☐Well-being

☐Engagement/Presence of Parties ☐ ICWA

☐Other:_____________________________________________________

☐Other:______________________________________________________

6. Self-Assessment – Capacity

We would like you to assess your current capacities related to knowledge, skills, resources, and collaboration by responding to the following 3 sets of questions.

1. Please indicate your level of agreement to the following statements.

	Strongly Disagree	Disagree	Somewhat Disagree	Neither Agree nor Disagree	Somewhat Agree	Agree	Strongly Agree
I have a good understanding of CQI.	☐	☐	☐	☐	☐	☐	☐
I understand how to integrate CQI into all our work.	☐	☐	☐	☐	☐	☐	☐
I am familiar with the available data relevant to our work.	☐	☐	☐	☐	☐	☐	☐
I understand how to interpret and apply the available data.	☐	☐	☐	☐	☐	☐	☐
The CIP and the state child welfare agency have shared goals.	☐	☐	☐	☐	☐	☐	☐
The CIP and the state child welfare agency collaborate around program planning and improvement efforts.	☐	☐	☐	☐	☐	☐	☐
We have the resources we need to fully integrate CQI into practice.	☐	☐	☐	☐	☐	☐	☐
I have staff, consultants, or partners who can answer my CQI questions.	☐	☐	☐	☐	☐	☐	☐

2. How frequently do you engage in the following activities?

	Never	Rarely	Sometimes	Often	Always
We use data to make decisions about where to focus our efforts.	☐	☐	☐	☐	☐
We meet with representatives of the child welfare agency to engage in collaborative systems change efforts	☐	☐	☐	☐	☐
We evaluate newly developed or modified programs/practices.	☐	☐	☐	☐	☐
We use evaluation/assessment findings to make changes to programs/practices.	☐	☐	☐	☐	☐
CQI is integrated into all our projects.	☐	☐	☐	☐	☐

3. Please review the descriptions of the different levels of collaboration. Using the scale provided, please indicate the extent to which you currently interact with each other partner identified below.

		Networking 1		Cooperation 2		Coordination 3		Coalition 4		Collaboration 5
Relationship Characteristics		--Aware of organization --Loosely defined roles --Little communication --All decisions made independently		---Provide info to each other --Somewhat defined roles --Formal communication --All decisions made independently		--Share information and resources --Defined roles --Frequent communication --Some shared decision making		--Share ideas --Share resources --Frequent and prioritized communication --All member have a vote in decision-making		--Members belong to one system --Frequent communication is characterized by mutual trust --Consensus is reached on all decisions
	No Interaction at all 0		Networking 1		Cooperation 2		Coordination 3		Coalition 4		Collaboration 5
State Child Welfare Agency	☐		☐		☐		☐		☐		☐
Tribal Child Welfare Agencies	☐		☐		☐		☐		☐		☐
Tribal Courts	☐		☐		☐		☐		☐		☐
Department of Education/ School	☐		☐		☐		☐		☐		☐
Law enforcement	☐		☐		☐		☐		☐		☐
Juvenile justice agency (e.g., DOJ)	☐		☐		☐		☐		☐		☐
Behavioral/mental health	☐		☐		☐		☐		☐		☐
Substance abuse/addictions management agency	☐		☐		☐		☐		☐		☐
Other:____________________	☐		☐		☐		☐		☐		☐
Other:____________________	☐		☐		☐		☐		☐		☐

7. Timeliness Data & Performance Measurement

The purpose of asking all the CIPs to report on timeliness measures has been to prompt you to identify available data, examine how you are currently doing, and make comparisons to how you have done in the past on specific measures. The goal is to help you identify where you are and encourage you to use data in a meaningful way in your systems change efforts. As such, we have restructured the timeliness requirements so that you can still report on the timeliness measures but have the option to report on other measures that you have found particularly meaningful in your work.³

1. Timeliness. Provide a narrative below describing where you are getting data and how you are calculating the timeliness measures you report. What is your universe of cases (e.g., what is your sample, exit or entry cohort, etc.)? Is the data from the agency (e.g., SACWIS), from a court case management system (e.g., Odyssey) or from another source? Do you have any concerns with the accuracy of the data?

	Baseline Measure (FY 2013)	FY 2014	FY 2015	CIP Projects Targeting Measures (if applicable) [If this measure was targeted by an intervention (e.g., efforts made to improve timeliness), please list the project or activity here]
Required Timeliness Measures
4G. Time to First Permanency Hearing
4H. Time to Termination of Parental Rights Petition
4I. Time to Termination of Parental Rights
4A. Time to Permanent Placement
Optional Measures
Time to Reunification
Time to Adoption
Time to Guardianship
Time to Emancipation
Time to Subsequent Permanency Hearings
1B. Percentage of Cases that Re-enter within 1 year

2. Other Measures. What other measures do you collect that you find particularly useful?

Do you currently or have you recently collected any data on quality legal representation or quality court hearings that you would be willing to discuss or share? If yes, please explain?

Appendix A: Definitions

Definitions of Evidence

Evidence-based practice – evidence-based practices are practice that have been empirically tested in a rigorous way (involving random assignment to groups), have demonstrated effectiveness related to specific outcomes, have been replicated in practice at least one, and have findings published in peer reviewed journal articles.

Empirically-supported- less rigorous than evidence-based practices are empirically-supported practices. To be empirically supported, a program must have been evaluated in some way and have demonstrated some relationship to a positive outcome. This may not meet the rigor of evidence-base, but still has some support for effectiveness.

Best-practices – best practices are often those widely accepted in the field as good practice. They may or may not have empirical support as to effectiveness, but are often derived from teams of experts in the field.

Definitions for Work Stages

Identifying and Assessing Needs – This phase is the earliest phase in the process, where you are identifying a need to be addressed. The assessing needs phase includes identifying the need, determining if there is available data demonstrating that this a problem, forming teams to address the issue.

Develop theory of change—This phase focuses on the theorizing the causes of a problem. In this phase you would identify what you think might be causing the problem and develop a “theory of change”. The theory of change is essentially how you think your activities (or intervention) will improve outcomes.

Develop/select solution—This phase includes developing or selecting a solution. In this phase, you might be exploring potential best-practices or evidence-based practices that you may want to implement as a solution to the identified need. You might also be developing a specific training, program, or practice that you want to implement.

Implementation ­– the implementation phase of work is when an intervention is being piloted or tested. This includes adapting programs or practices to meet your needs, and developing implementation supports.

Evaluation/assessment – the evaluation and assessment phase includes any efforts to collect data about the fidelity (process measures: was it implemented as planned?) or effectiveness (outcome measures: is the intervention making a difference?) of the project. The evaluation assessment phase also includes post-evaluation efforts to apply findings, such as making changes to the program/practice and using the data to inform next steps.

1 Definitions for evidence-based, empirically-supported and best-practices are available in the appendix.

2 A description of each stage of work is available in an appendix to this document.

3 The OJJDP Toolkit that includes these performance measures can provide additional guidance and is available online at: http://www.ojjdp.gov/publications/courttoolkit.html

8