Cross-Center Evaluation: Leadership Interview – States and Territories

PURPOSE: The purpose of the interview is to obtain information from State Child Welfare Directors regarding factors that influence their decisions to engage in services with the CB-supported Capacity Building Collaborative (specifically the Center for States); perceptions of the capacity building services received; and satisfaction with the Center’s services.

OMB #: 0970-0XXXX

Expiration Date: XX/XX/XXXX

PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) STATEMENT OF PUBLIC BURDEN: This collection of information will be used to obtain information from state child welfare directors regarding services received from the Capacity Building Collaborative. Public reporting burden for this collection of information is estimated to average 40 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. This is a voluntary collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #: 0970-XXXX, Exp: XX/XX/XXXX. If you have any comments on this collection of information, please contact Serena L. Williams, ACF, Administration on Children, Youth and Families by e-mail at [email protected].

Evaluation of the Child Welfare Capacity Building Collaborative

State Child Welfare Leadership Interview

The purpose of the interview is to obtain information from State Child Welfare Directors regarding factors that influence their decisions to engage in services with the CB-supported Capacity Building Collaborative (specifically the Center for States); perceptions of the capacity building services received; and satisfaction with the Center’s services.

Interview Code: S___

Introduction:

Good morning/afternoon. My name is ______________. I am calling from James Bell Associates, an evaluation firm based in Arlington, VA. As you may know, James Bell Associates and ICF have been contracted by the Children’s Bureau to conduct an independent evaluation of the Children’s Bureau’s Capacity Building Collaborative, which includes of the Center for States, Center for Tribes, and Center for Courts.

Directions to interviewer – please inform the interviewee:

This evaluation addresses the role of the Capacity Building Collaborative in helping States, Tribes, and Court Improvement Programs (CIPs) to support and enhance the capacity of their child welfare systems. The Leadership Interview is an important component of the evaluation. We are conducting these telephone interviews with all State child welfare directors as well as Tribal directors and CIP directors who receive capacity building services from the Centers.

I am going to ask you about the factors that may influence your decisions to use the services of the Capacity Building Collaborative and your experiences with the services you have received from the Center for States.

Your participation in this interview is voluntary but important to this evaluation. We anticipate that our discussion with you today will take approximately 40 minutes. Your privacy is important to us. As we have noted in scheduling this interview with you, in order for us to ensure that we accurately capture everything, we would like to audio record this interview. The recording will be destroyed as soon as we transcribe it and we will remove all identifying information from the transcript. Responses that you provide will be combined with answers from other respondents and will not be associated with your agency or you as an individual respondent. Do we have your permission to audio record this conversation? [Interviewer: Wait for the response] Do you have any questions or concerns before we begin?

I will now start the recording. [Interviewer: Begin audio recording]

1. RESPONDENT BACKGROUND

1. What is your official position title?

2. How long have you been in this position?

_______years or _____months

B. THE CAPACITY BUILDING CENTER

First a question about the Capacity Building Center for States.

1. How familiar are you with the Capacity Building Center for States and its services? Please describe.

C. ASSESSMENT AND WORK PLANNING

Interviewer: If the State did not participate in assessment, move to Section D.

1. How useful, or not, was the assessment to your agency?

   1. Did your agency’s understanding/awareness of the strengths and gaps in your CW system increase through the assessment process? Please briefly explain.

   2. Did your agency’s capacity to self-assess its capacity building needs increase through the assessment process? Please briefly explain.

   3. How accurately did the assessment summary identify your state agency’s capacity building needs?

2. Was a work plan developed as a result of the assessment?

If no, skip to C3

1. In your opinion, how useful, or not, was the work plan?

2. Did the work planning process help you to achieve things that you would not have achieved without the Center?

3. Do you have additional thoughts that you would like to share regarding your agency’s experiences with the assessment process or the work plan that was developed with the Center?

D. WORKING WITH THE CAPACITY BUILDING CENTER

Now, let’s talk about the services that your agency received from the Capacity Building Center for States.

D1.a. [Ask of States that received services from the Centers] According to the Children’s Bureau’s records, the services that your agency received or participated in are listed in the document I sent you. Is that list of services accurate or is something missing? [skip to D2.a.]

Prefill from CapTRACK Tailored Services Records 

Assessment and workplans
Assessment service start	Assessment service end	Assessment hours	Workplan service start	Workplan service end	Workplan hours
Projects
Service start date	Date closed	Service hours	Project title		Project related to CFSR/PIP? (Y/N)
Other tailored services
Year	PIC hours	CFSP hours	FFPSA hours
CY20
CY19

D1.b. [Ask of States that did not receive services from the Centers] According to the Children’s Bureau’s records, your agency has not received capacity building services from the Capacity Building Center for States. Is this correct? [skip to D2.b.]

D2.a. [Ask of States that received services from the Center] What were the key factors that influenced your decision to engage with the Center for States to utilize the Center’s services? Please describe. [skip to D3.]

D2.b. [Ask only of States that did not receive services from the Center] What were the key factors that influenced your decision to not use the Center’s services?

3. Let’s take a look at the list of potential factors that is included as item D3 in the interview questions summary document I sent you. Which, if any, of these factors also influenced your agency’s decision to use (or not to use) the Center’s services.

Affected decision to use services (+ or -)	Factors	Comments
	CFSR findings or development of the PIP
	Regional Office recommendation
	Federal law or policy change or requirements
	Center’s outreach efforts
	Prior relationship with the Center
	Timeliness in which services can be received
	Center staff’s level of knowledge and skills
	Cultural competency of the Center staff
	Usefulness of the Center’s products/materials
	Child welfare incident in the State (such as a child fatality)
	Availability of free resources from the Center
	Availability of State agency staff time
	State child welfare agency leadership
	State/local law or policy change
	Lawsuit/legal settlement
	COVID-19 pandemic

E. SERVICES RECEIVED AND PROGRESS TOWARD OUTCOMES

The next questions (section E in the interview guide that we sent you) pertain to: [Prefill from item D1 brief description(s) of the tailored services]

1. First, could you describe the current status of your State’s work with the Capacity Building Center for States?

2. In your opinion, what are the primary outcomes that your State expected to achieve from the work with the Center?

(Interviewer: Probe also)

1. Are there any additional primary outcomes that your State expected to achieve?

Now, let’s talk about the progress that has been made toward the primary outcome(s).

[Interviewer: Ask questions E3: A, B, C and repeat the questions for additional outcomes if respondent highlighted more than one outcome

3. Outcomes:

1. Please describe the progress that has been made toward [outcome from question E2].

2. To what extent did the services provided by the Center contribute toward this outcome?

3. Were there other key factors that also contributed toward this outcome?

4. Was your work with the Center intended to result in any practice changes in your child welfare program?

   1. If yes  In your opinion, are your agency’s practices changing or improving in any way? Please describe.

   2. What do you attribute these changes to?

   3. And are there any changes in outcomes for the children and families served by your child welfare system?

   4. If yes  Please describe. (Interviewer: Probe also for what evidence they have of these changes)

      1. In your opinion, what contributed to these changes?

5. Were there any key factors that may have hindered your agency from making progress toward achieving the intended outcomes? If so, please describe. [e.g., organizational or agency factors, community factors, other environmental/contextual factors]

   1. Has the progress toward outcomes been impacted by the COVID-19 pandemic? If yes, please describe.

6. Overall, how would you describe the quality of the services/materials your agency received from the Center?

7. Based on the services your agency received, how would you describe the expertise of the Center liaisons?

1. And how would you describe the expertise of the Center consultants.

8. In your opinion, what have been the key benefits of working with the Center?

9. Based on your work with the Center, do you have any suggestions for improving the Center’s services?

IF tailored services have DEI focus or component, probe also:

Do you have any suggestions for improving the Center’s services and support on issues related to diversity, racial equity, and inclusion (including tribal sovereignty) and advancing equity and support for underserved communities?

F. STATE’S EXPERIENCE WITH THE CENTER FOR STATES

Interviewer:

• States that received tailored services: Ask questions F1 through F10.

• States that did not receive tailored services but did participate in the assessment/work planning process: Ask F1 through F6, and F10.

• States that did not receive tailored services and did not participate in the assessment: Ask only F1 and F3.

Please provide your level of agreement with the following statements, using the response categories, from 1 to 6: Strongly disagree; Disagree; Somewhat disagree; Somewhat agree; Agree; Strongly agree. These response categories are also listed in your copy of the interview questions document.		1 Strongly disagree	2 Disagree	3 Somewhat disagree	4 Somewhat agree	5 Agree	6 Strongly agree	Do not know
F1.	My agency is satisfied with the level of accessibility of the Center for States
F2.	When working with the Center for States on issues related to my child welfare system, my agency has been satisfied with the frequency of communication
F3.	My agency is comfortable disclosing areas of concern or weaknesses of our agency to the Center for States’ staff
F4.	When working with the Center for States, my agency’s experience is that our agency plays an active part in decision-making
F5.	My agency is satisfied with the level of follow-through from the Center for States
F6.	Overall, my agency is satisfied with our relationship with the Center for States
F7.	Overall, the services that the Center has provided to my agency reflect an understanding of how my child welfare system operates
F8.	The services provided by the Center were appropriate and aligned with my child welfare system’s needs
F9.	The services provided by the Center took into account the characteristics and unique context of my State

G. IN CONCLUSION

G1. Do you have any additional comments you would like to share regarding CB’s Capacity Building Collaborative or your experiences with the services they provide?

THANK YOU

WE GREATLY APPRECIATE YOUR PARTICIPATION!