Form 1 Assessment Protocol

Scheduling/ Pre-Interview Guidance:

• You must have reviewed and be very familiar with the state profile, the CFSP, and supplemental documents (such as the latest CFSR, self assessment, PIP, fiscal reports, previous assessments, and targeted plans). Plan to take a copy of the CFSP with you on site.

• You will have multiple informative conversations with your Regional Office partners prior to conducting assessment interviews in order to have a thorough understanding and historical perspective of the state, based on the regional team’s knowledge and working relationships with the state.

• Based on information obtained via the state profile, please see ‘materials needed’ listed on Supplemental Agency Information to inform request to states for documents that could be sent prior to general assessment interviews. They might help in providing context or in giving suggestions on prompts or questions specific to the jurisdiction.

• As you prepare for your interview, please read through the items listed on the Supplemental Agency Information document to help inform and give context to the additional types of information we are trying to draw out and listen for during the interview.

• You and your Regional Office colleagues will make a plan to contact the Child Welfare Director of the state. During the conversation, you will explain the purpose of the interviews: to gather information to help assess the state’s capacities in order to work together to develop a plan for building capacities. It is as a result of this assessment that services will be provided.

• All states interested in service from the Capacity Building Center for States will participate in the general assessment process. The assessment is intended to be a service and a support to the state as the state considers and evaluates new directions and innovations for child welfare. All services are linked to building capacities identified in the assessment. Eventually, the assessment will support the CFSP and APSR planning process. The interview and document review are to gather information that will be formulated into a narrative assessment with recommendations for next steps including potential brief services or referral for intensive services. The assessment will be shared with the Child Welfare Director and his/her designees in order to develop a collaborative plan for next steps. Inform the Child Welfare Director of the time estimates related to the state’s staff. No more than 8 hours of state staff time will be needed to prepare, followed by 16 hours on site of state staff time, and no more than 8 hours of follow up time for state staff. No interview will take more than 2 hours per person, and the interviewer will only be talking with staff designated for interviews. The questions will be sent in advance so that everyone can be prepared.

• The Child Welfare Director will determine all points of contact. An agency senior administrator or designated point of contact will assist with identifying the appropriate people to participate in the assessment interviews. Each individual interview will be less than 2 hours. We will make a request to speak with 5-6 people who can help inform us in the following program areas (some people may be able to speak to more than one program area):

  • Agency structure and operations

  • Budget/finance

  • Data

  • Policy and practice standards

  • QA/QI

  • Human resources/staff development

  • Other as needed as determined by Child Welfare Director and based on specifics of the jurisdiction

• Interviewees will be provided with a copy of the assessment protocol questions prior to the interview to better help them prepare.

• As you prepare for the interview, keep the end in mind which is an assessment of the following capacities: Resources, Infrastructure, Knowledge and Skills, Culture and Climate, and Engagement and Partnerships. The information gathered here should inform the determination of next steps in building foundational or innovation specific capacities or in delving deeper into problem analysis. The answers to these questions will be analyzed and compiled into a narrative assessment report that includes recommendations for next steps.

Tips for the Interview:

• Remember this is an opportunity to begin to build relationships with agency staff, and is intended to be conducted in a conversational style. It is the beginning of what will be ongoing discussions.

• Based on the program areas the interviewee will be covering, please review list of capacity area prompts that are relevant before the interview.

• Thank the interviewee for taking the time to talk with you.

• Explain you are from the National Capacity Building Center for States and one of the partners of the Children’s Bureau who is available to serve the state. The partnership will work together to assess the state’s needs, and determine the capacity needs that could be developed.

• Explain the assessment is designed to determine the areas of strengths and challenges for the state in order to provide resources to help address the challenges. This is to be a discussion so that we can learn the state’s perspective and help think through problems and potential innovations.

• This interview should take no more than two hours per individual.

• Explain that we may be back in touch if we have follow up questions and thank them for their ongoing support.

• Answer any questions about the process before getting started.

• Share just a few sentences on what brought you into the work, what you like about it, and 1-2 details about your background. Include your familiarity with their data and the ongoing partnership with the regional staff.

• Ask interviewee to share background information about themselves: how long they’ve worked there, other roles held, overall impressions about the agency/jurisdiction.

• Close the interview with a discussion of next steps and what the state can expect.

• If there are questions that the interviewee can’t answer, ask for the name and contact information of the person they suggest to provide the information which would then be communicated back to the state lead for confirmation of next steps.

• If documents are referenced that might be helpful later, ask for a copy of the document to be sent to you.

• Please be sure to capture information in such a way that details and description are available to inform the analysis that will be produced subsequent to the assessment.

These 11 questions are designed to facilitate a discussion of capacities in order to help determine the state’s needs in developing capacities. The state will determine who should be involved in answering the questions. The time for the state to prepare should be approximately 8 hours pre-site visit. Time on site should be no more than 16 hours. Follow up time after the site visit should be approximately 8 hours. These estimates are averages and inclusive of all staff time involved aggregated as a total estimate. This estimate is not per person.

Each question should be asked of every state/territory. The bullet lists under the questions are prompts to be probed if the items are not referenced in the discussion following each question. The questions are intentionally broad to assure each state/territory has the ability to share unique and individual answers and to engage in discussions that are meaningful to the individual state/territory.

GENERAL ASSESSMENT PROTOCOL

1. What is working well within child welfare in your state? How did you come to this determination?

   1. Projects or initiatives

   2. CW demonstration activities

   3. Initiation of change efforts

   4. Leadership

   5. Workforce

   6. Fiscally sound/creative

   7. Program support

   8. Collaborations

   9. Use of data

2. What areas of concern did you identify in the CFSP (let’s look at it together) and how did you determine these were problematic? Are there areas you are unsure of in your analysis? Are there other areas that cause you concern?

   1. Programs

   2. Partner relationships

   3. Outcomes

   4. Staff

   5. Infrastructure

   6. Budget

3. Can you describe your current efforts in addressing the above concerns and where you are in your developmental efforts? Do you feel you are on track?

   1. Identifying resources and teams

   2. Planning and designing the intervention

   3. Implementing: staging, piloting

   4. Monitoring: service delivery is underway

   5. Evaluating and transitioning to sustainability

4. Can you please tell me about any distinguishing characteristics of your agency/region and the impact these characteristics have on the child welfare program?

   1. Immigrant populations

   2. Resettlements

   3. Any particular groups over or under represented as it related to the CW population

   4. Any jurisdictions within the state that are unique (size, population, structure, influence on the rest of the state)

   5. Lawsuits or Consent decrees

5. Can you please tell us about your workforce and what aspects of infrastructure/hiring/workload/education are supporting or preventing the workforce from addressing the main needs and goals you identified in your CFSP?

   1. Average tenure

   2. Educational requirements

   3. Turnover/vacancy rates

   4. Average caseloads for various units/departments

   5. Promotional process

   6. Strengths

   7. Challenges

   8. Developmental opportunities

   9. Training system

   10. Performance measures

6. How would you describe your overall organizational structure, culture and climate? Is there anything about the culture or climate that contributes to your concerns expressed above?

   1. Norms

   2. Expectations

   3. Morale

   4. Communication

   5. Relationships among staff

   6. Relationships between counties/regions

   7. Structure: county vs. state administered, program structure

   8. Privatization

   9. Access to supports and resources

7. What current initiatives, special projects, or priorities are you focusing on now and what does this focus look like?

   1. Process for initiation

   2. Assigned lead/ involved parties

   3. Desired outcome

   4. Metrics for monitoring

   5. Theory of Change and Logic Model

   6. Timeline

   7. Statewide or targeted

8. How is data collected and used in your system to make decisions, drive change, and assess performance?

   1. Regular reports

   2. Types of data collected (qualitative and quantitative)

   3. Access at which levels of staff and decision makers (internal and external)

   4. Types and reliability of data systems used

   5. Used to drive decision making

   6. Utilization of CQI

   7. Methodology regarding assessments and analysis

   8. Case review capacity

   9. Feedback loops

9. How do you manage change in your organization and how successful have you been in the past with this approach?

   1. Phased approach

   2. Messaging

   3. Work groups

   4. Fiscal supports

   5. Planning and research

   6. Policy implications

10. Which of your partner organizations represent strong, supportive collaborations in achieving outcomes and which represent barriers in achieving outcomes? Why?

    1. Unions

    2. Foster care agencies

    3. Courts

    4. Mental health

    5. Juvenile justice

    6. Education

    7. Group care

    8. Tribes

    9. Other

11. In what way do you see the Center being helpful in your change efforts?

    1. Assisting with implementation

    2. Exploring problems

    3. Other

Note to interviewer: Please read through this list of Supplemental Agency Information prior to your interview to give an idea of the kinds of information we are trying to draw out and listen for during the interview. Be mindful that this information is analyzed by capacity type and type of person being interviewed, so the supplemental information being gathered may differ based on the person being interviewed and their role within the agency. By the end of the assessment, the liaison should have gathered as much as this information as possible and/or relevant.

Supplemental Agency Information

CW Organizational Capacity Area: RESOURCES (Adequacy and stability of resources – fiscal, staffing, materials, facilities, equipment, and access to information)
Information Source	Supporting Documents	Information Type
Administrators HR staff Staff development IT staff Fiscal staff	Facilities reports Staffing reports Training plans Standard data reports	Types of available material resources Technology Facilities Supplies Transportation Types of available operational resources Trainings Staff development Other
CW Organizational Capacity Area: INFRASTRUCTURE (Policies, processes, and operational structures)
Information Source	Supporting Documents	Information Type
Administrators Fiscal staff HR staff	Org charts Decision making tools Practice Models Theories of Change Logic Models Organizing frameworks Evaluation tools Targeted plans Performance standards	Primary fiscal resources Federal funds (IV-E, IV-B, Chaffee) State/local funds Grants (types, source) Private funding Flexible funding Other Fiscal incentives or disincentives Adoption Other Well-funded program areas Under-funded program areas Description of Practice Model System barriers impeding achievement of desired outcomes Low staff morale Budget constraints High turnover Legal issues Media pressures Other Methods of evaluation Internal External Caseworker standards Programmatic Frequency Responsible parties Types of agency performance improvement mechanisms/processes in place Internal Statewide Federal CQI Other
CW Organizational Capacity Area: KNOWLEDGE & SKILLS (Knowledge, skills, expertise- practice knowledge, analytic and evaluative abilities, and knowledge of change process)
Information Source	Supporting Documents	Information Type
Management Administrators HR staff Staff development QA staff	Training plans Practice Models Evaluation/ Assessment plans Theories of Change Logic Models Performance standards	Types of performance assessment used for staff Types of staff development available Internal External New Hire Ongoing Eligibility State responsibilities regarding training Local responsibilities regarding training Agency approach to continuous improvement
CW Organizational Capacity Area: CLIMATE & CULTURE (Values, norms and attitudes; commitment to agency practice)
Information Source	Supporting Documents	Information Type
Administration Management Staff development Program staff	Mission statement Vision statement Organizational goals Child and family outcomes data	Examples of successful change efforts Focus area Factors supporting success Examples of unsuccessful change efforts Focus area Factors inhibiting success Staff building activities Frequency Participants Lead
CW Organizational Capacity Area: ENGAGEMENT & PARTNERSHIP (Collaboration and cooperation within the agency, between the agency and external partners, and with communities)
Information Source	Supporting Documents	Information Type
Administration Managers	Communication plans MOU’s Newsletters	Communication within the agency Mechanisms Frequency Feedback loops Gauge of effectiveness Communication external to the agency Existence of communication plans Frequency Mechanisms Feedback loops Internal stakeholder involvement in key activities Planning Implementation Evaluation Decision making External stakeholder involvement in key activities Planning Implementation Evaluation Decision making Relationships with local tribes Communication Collaboration Agency engagement with consumers Families Youth Elders Community

8