Center for States (CBCS) Evaluation: Jurisdiction Interview Protocol

Jurisdiction Interview Protocol

PURPOSE: Participants will include all project leads for approximately 30 projects that are receiving intensive support from the Center for States. For each project, the lead staff from the child welfare agency will be interviewed once, to gather feedback on their experience in working with the Center in general, collaborating around service provision or feedback on a specific service or set of services in which they participate.

OMB #: 0970-XXXX

PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) STATEMENT OF PUBLIC BURDEN: The purpose of this information collection is to gather basic registration information on capacity building services to better meet the needs of child welfare professionals. Public reporting burden for this collection of information is estimated to average 60 minutes per respondent, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. This is a voluntary collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number. The control number for this project is 0970-XXXX. The control number expires on XX/XX/XXXX. If you have any comments on this collection of information, please contact Beth Claxon, ACF, Administration on Children, Youth and Families by e-mail at [email protected].

Expiration Date: X/XX/XXXX

Tailored Services Intensive Projects

Interview Note taking sheet: JURISDICTION STAFF

• Interviewer:

• Note taker:

• Interviewee/State:

• Date/Time:

Highlights= summarize (not necessary to read word for word)

Bold= read word for word

Italics= probes

Prior to the Interview

In the days leading up to the interview, the interviewer will review the following:

1. Project’s work plan and close out report to help create effective and more specific probes.

2. Project milestones and capacity dimensions identified in the workplan and final report to assist in asking about Center impacts.

3. Project status (i.e., normal close out, ended early, not implemented, the project is continuing), as interview questions and probes will need to be tailored to the project status.

The interview coordinator will share the workplan, final report, and close out report with facilitators and notetakers prior to the date of the interview. These are also available in CapTRACK.

Overview

Briefly discuss the purpose of the interview:

The purpose of our conversation is to gather feedback about your experiences with the Center for States services for [insert project name]. Your contribution to the evaluation effort is extremely valuable and will be used to improve future services.

Convey to each interview participant our privacy policy:

(1) the interview is voluntary; (2) you can decline to answer any questions, or you can stop participating in the interview at any time; (3) Data from this interview will be held private by the evaluation team. The Information from today’s interview will be reported and may be shared with the Children’s Bureau and the Capacity Building Collaborative’s 3^rd party evaluator. When reported this information will be aggregated across all states and your name will not be associated with it. In some instances, however, the information you provide about the success and outcomes of the project might be tied to the project itself in our reporting.

Ask permission to tape record the interview:

To ensure we capture the discussion accurately and completely, I would like to obtain your permission to audio record the session. This recording will only be accessed by our evaluation team. If you choose not to have the interview recorded, we take notes. Your name will not be included in any reporting.

Will you allow us to record this interview?

The interviewer asks the notetaker to start the recording.

Thank you, the recording has started.

Ask if they have any questions for you before you begin.

Ask them to provide their verbal consent to participate in the interview.

Do you have any questions before we begin?

Introduction

If applicable, clarify the status of the project and mention whether they may have already participated in an earlier project interview.

[CONTINUATION PROJECTS, FIRST INTERVIEW] I understand that the project is continuing onto a new work plan. However, for today’s interview, we would like you to focus on the most recent workplan that covered [insert date range from last workplan] (because we will be doing separate interviews for the current workplan).

OR

[CONTINUATION PROJECTS, PRIOR INTERVIEWEE] We recognize that you may have already participated in an interview related to this project for a previous workplan. However, we also conduct

interviews after completing each workplan. For today’s interview, we would like you to focus on the most recent workplan that covered [insert date range from last workplan]. We appreciate your time and perspective on this interview.

OR

[END OF PROJECT] We recognize that this project has ended. For today’s interview, we would like you to focus on the workplan that covered [insert date range from last workplan]. We appreciate your time and perspective on this interview.

Questions

1. Can you begin by briefly describing your role at the agency?  What was your involvement with Center support and role on the project?

2. To what extent do you understand the overarching goal(s) of this project? Would you say that you…

• Fully understand project goals

• Mostly understand project goals

• Partially understand project goals

• Do not understand the project goals

3. According to the project workplan, the original desired change or primary goal was [insert Jurisdiction's Desired Changes in Child Welfare Practice from workplan]. To what extent did the project make progress towards achieving this desired change?

4. What has been your experience with the Center’s services [or for continuing projects- What has been your experience with the Center’s services so far]?

Probe about these Practice Model phases…

Engagement:

5. To what extent did the Center engage all necessary stakeholders to actively participate and collaborate (e.g., including federal partners, university partners, community agencies, etc.)? Would you say that the Center…

• Fully engaged all necessary stakeholders

• Mostly engaged all necessary stakeholders

• Partially engaged all necessary stakeholders

• Did not engage all necessary stakeholders

Please explain your response.

6. To what extent was the Center team a “good fit” with your agency? Would you say that they were…

• Entirely a good fit

• Mostly a good fit

• Partially a good fit

• Not a good fit

Please explain your response.

7. To what extent did your agency feel that it had ownership of this project? Would you say that…

• They felt they had full ownership of the project

• They felt they had substantial ownership of the project

• They felt they had partial ownership of the project

• They felt they had no ownership of the project

Please explain your response.

7. To what extent do you think your agency had authority to make project decisions? Would you say that…

• Your agency had full authority to make project decisions

• Your agency had considerable authority to make project decisions

• Your agency had partial authority to make project decisions

• Your agency had no authority to make project decisions

Please explain your response.

7. To what extent do you think your agency was ready to take on the project work? Would you say that…

• Your agency was fully ready to take on the project work

• Your agency was mostly ready to take on the project work

• Your agency was partially ready to take on the project work

• Your agency was not ready to take on the project work

Please explain your response

7. To what extent was there buy-in from leadership? Would you say that leadership had…

• Fully bought-in to the project

• Mostly bought-in to the project

• Partially bought-in to the project

• Not bought-in to the project

Please explain your response

Working as a team:

7. To what extent do you agree with the following statement, “The Center and my agency effectively worked together as a team.” Would you say that you “agree”, “somewhat agree”, “somewhat disagree” or “disagree” with this statement?

Please explain your response.

Work Planning:

7. How helpful was Center support in thinking about what you wanted to achieve and how you would know if you did achieve your goals? What about it was helpful or not? What could have been improved?

Tracking and Adjusting:

7. How well is the Center helping your state to identify what data were needed and how to secure those data? Can you think of an example of how the team has used data to determine progress and inform planning and decision making?

7. Now I would like you to describe in what ways, if any, Center services benefitted your jurisdiction. [After general description of benefit or no benefit, probe for EVERY improvement area below.]

Did Center support lead to improvements, in:

1. Agency staff knowledge, awareness, or skills? If so, please provide an example. Were these changes you saw right away or later in the project?

2. Agency policy? If so, please provide an example. Were these changes you saw right away or later in the project?

3. Agency staff practices? If so, please provide an example. Were these changes you saw right away or later in the project?

4. Organizational infrastructure (e.g., training systems, staffing, CQI systems)? If so, please provide an example. Were these changes you saw right away or later in the project?

5. Outcomes for children and families (e.g., increased permanency for children, reduced time in foster care, reduced number of placements, reduced recurrence of CAN, etc.)? If so, please provide an example. Were these changes you saw right away or later in the project?

6. Other types of improvements, outcomes, or impacts? If so, please provide an example. Were these changes you saw right away or later in the project?

If any impacts mentioned, use additional probes:

7. What factors do you think helped to facilitate the impacts you described (list impacts mentioned by respondent)?

8. Which of the impacts you just described will be sustained?

• Probe: How well is the Center helping prepare your agency to sustain capacity for this work?

7. What factors, if any, impeded project progress?

7. Were you involved with onsite Center support prior to the Center moving to 100% virtual service delivery since COVID-19 pandemic began? If so, how well did the Center shift from primarily onsite to all virtual service delivery?

• Probe: In what ways, if any, did the shift from onsite to virtual affect your project?

7. What is the Center doing well with virtual service delivery? Do you have any suggestions for ways the Center can improve its virtual support?

• Probe: Do you have any suggestions for the frequency and length of virtual meetings?

7. END-OF-PROJECT VERSION: How well did the Center help prepare your state to sustain capacity for this work?

Probe if answer is negative: What could have been done differently? [For continuing projects: How did the Center help your jurisdiction prepare for sustaining the progress made thus far with this project?]

FOLLOW-UP VERSION: How is the jurisdiction sustaining earlier progress/capacity? 

Now I have a few questions about the organizational change and implementation process that the Center used when supporting this project. As you might already know, change management is an evidence-based process for organizational change and implementation that the Center has organized according to the following five phases.

Phase I - Identify and Assess Needs or Opportunities 

Phase II - Develop Theory of Change 

Phase III - Select and Adapt/Design Intervention 

Phase IV - Plan, Prepare, and Implement 

Phase V - Evaluate and Apply Findings 

7. How familiar are you with the organizational change and implementation process used by the Center?

7. Has your agency considered applying organizational change and implementation process in another effort? If so, how? If not, what would have helped your agency implement an evidenced informed change and implementation process?

7. To what extent, if at all, did your agency’s capacity to evaluate practice change improve as a result of support from Center staff?

1. [If the interviewee reports that their capacity to evaluate practice change improved as a result of support from Center staff, ask] In what ways did Center staff help your agency to improve its capacity to evaluate practice change?

7. [If time permits, ask this question]: What were the most helpful activities/strategies used by Center staff in working with your jurisdiction on this project? What were the least helpful?

Possible probes for strategies: Toolkits; assessments, involving frontline staff; cross collaboration with different stakeholders, coaching. The interviewer can also refer to the workplan for specific examples.

Types of strategies:

Most helpful:

Least helpful:

7. What lessons or recommendations do you have for other states beginning to work with the Center?

7. Are there other comments you would like to provide about your experience with the Center for States? 

Thank you for taking the time to share your perspectives with us!