Form 1 Agency Results Survey

Cross-Site Evaluation of the Children’s Bureau’s Child Welfare Technical Assistance Implementation Centers and National Child Welfare Resource Centers

OMB No.: XXXX-XXXX

Expiration Date: 00/00/0000

Agency Results Survey

For internal uses only:

State/Tribal ID Number: __________________________________________________

Respondent Name:______________________________________________________

Position Title:___________________________________________________________

Interviewer:_____________________________________________________________

Date:__________________________________________________________________

INSTRUCTIONS FOR INTERVIEWER:

For items noted as PREFILL, the interviewer will need to obtain this information prior to conducting the interview. This information may be gleaned from the Children’s Bureau’s technical assistance tracking system, semi-annual progress reports, or agency websites.

Introduction

Hello. My name is ______________. I am calling from James Bell Associates, an evaluation firm based in Arlington, VA. As you may know, James Bell Associates and ICF International have been contracted by the Children’s Bureau to conduct an independent evaluation of the Children’s Bureau’s Implementation Centers and National Resource Centers. This evaluation addresses the role of technical assistance in helping states and tribes to improve their child welfare systems. We want to learn more about the technical assistance activities and services that have been provided to [STATE/TRIBE] to help improve the child welfare system. For purposes of this evaluation, we define the child welfare system to include the child welfare agency, the courts and legal system, social service agencies, and other child-serving providers. Your participation is critical to this evaluation. We expect that the interview will take about 60 minutes. Your privacy is important to us. Information you provide during the interview will be combined with answers from other agencies and will not be associated with States or individual respondents.

Do you have any questions about what I’ve explained to you before we begin? I would like to first start out by asking you some basic information on your background and history with the agency.

First we need some information on your background and history with the agency.

1. Background

1. What is your position title?_____________________________________________

2. How long have you been in this position?

_______years or _____months

3. How long have you been with the agency?

_______years or _____ months

4. What other positions have you held within the agency?

2. Characteristics of the Child Welfare System

Now, we would like to clarify some background information on the child welfare system overall.

1. In [STATE/TRIBE], the name of the agency where child welfare resides is [Prefill].

Is this correct?

Yes

No (The correct name of the agency is _______________________)

2. And your child welfare system is:

Prefill 		Tribally administered/governed
Prefill 		County Administered
Prefill 		State Administered

Is this correct?

Yes

No (The system is ___________________-administered)

3. Is the child welfare unit accredited or in the process of being accredited?

Yes  By whom?

No

4. Are any key operations within the child welfare system privatized?

Yes  What parts of the child welfare system are privatized?

Probe: Intake investigation, in-home services, case management, foster care, adoption

No

4. Is the child welfare system operating under consent decree or settlement agreement as a result of a class action lawsuit?

Yes  What are/were the terms of the consent decree or settlement agreement?

No

3. Organizational and Systems Change

We want to begin by reviewing any changes that may have occurred in [STATE/TRIBE]’s child welfare system over the last year. Later we will discuss the technical assistance that has been provided. Remember that when we ask about the child welfare system we mean the child welfare agency, the courts and legal system, social service agencies, and other child-serving providers

Interviewer Instruction: There may be more than one response to C1.

1. In the past year have there been any changes in the child welfare system, including organizational or systems change? We are interested in learning what changed in the child welfare agency, with the courts and legal system, and across multiple agencies., if applicable

Yes  What changed within the child welfare agency?

How did it change? Please describe.

Were these improvements? Yes No

Were these changes desired or planned Yes No

What changed with the courts and legal system?

How did it change?  Please describe.

Were these improvements? Yes No

Were these changes desired or planned? Yes No

What changed across multiple agency systems?

How did it change?  Please describe.

Were these improvements? Yes No

Were these changes desired or planned? Yes No

No changes occurred. SKIP TO C4 

2. Have the organizational and/or systems changes you described resulted in improved outcomes for the children and families served by the child welfare system? Note to Interviewer: Reference the changes noted in C1 above.

Yes  What improved outcomes for children and families have been achieved?

What evidence is available to document these improvements?

No SKIP TO C4

Interviewer Instruction: Ask C3 in Years 3 and 5 only. Reference the changes discussed in the Year 2 interview, then ask C3.

3. A1. What organizational and/or systems changes achieved previously has the child welfare system been able to sustain?

A2. What changes has the child welfare system been unable to sustain?

Why? Can you explain what happened?

4. There are many reasons why the child welfare system may or may not be able to achieve the organizational or systems change it desires. There are also many factors, both internal to the organization and external, including community or contextual factors, that can either hinder or help to achieve the intended change.

From your perspective, what are the key factors that have helped the child welfare system to achieve its desired changes in the last year?

5. What key factors hindered the child welfare system from achieving its desired changes in the last year?

6. To get a complete picture of factors influencing change in the child welfare system, please tell me if there are any other factors provided on the following list that may have helped or hindered the child welfare system’s ability to achieve desired changes in the last year.

Additional factors:

Interviewer instruction. For those factors cited, record whether the factor helped or hindered and any comments provided. Where noted, specify whether the organizational leader mentioned is the Child Welfare Agency Director, a Judge, etc.

Factors

CB T/TA Provider Factors

Quality of the National Resources Centers’ technical assistance

Quality of the Implementation Center’s technical assistance

Intensity (frequency, duration, depth) of the NRCs’ technical assistance

Intensity (frequency, duration, depth) of the IC’s technical assistance

Organizational Agency Factors

Length of tenure of organizational leader. Specify:

Length of tenure of senior administrative/ managerial staff

Length of tenure of middle manager staff

Length of tenure of supervisor or frontline staff

Rate of staff turnover

Active involvement of management in the change

Active involvement of supervisors and/or frontline staff in the change

Leadership provided by organizational leader. Specify:

Availability of staff time

Staffing resources available

Financial resources available

Number of current change initiatives in agency

Priority level of the change initiative within agency

Organizational culture –attitudes, experiences, beliefs and values of the organization

Organizational climate or the work environment

Existing agency infrastructure

Environmental and Contextual Factors

[For Tribes only] Relationship with the State(s)

Adequacy of service array

Use of performance-based contracts

Agency’s relationship with partners

Change initiatives underway in partnering systems

Economic environment/condition

Lawsuit/legal settlement

State/local law or policy change

Federal law or policy change

Public perception of agency to fulfill mission to children and families

Use of technical assistance provided by those outside the Children’s Bureau’s T/TA Network:

• Private consultants

• Other Federal technical assistance providers

• Foundations

• University Partners

4. State/Tribal T/TA Needs

Now, about the [STATE/TRIBE’s] training and technical assistance needs.

D1. What are the most pressing issues or problems that the child welfare system has needed to address in the last year?

5. Familiarity with the Children’s Bureau T/TA Network

16. The Children’s Bureau sponsors free technical assistance to States and Tribes through the Child Welfare Training and Technical Assistance Network. Please indicate which members of the Network you are familiar with:

National Resource Centers		National Centers, Repositories, and Child Welfare T/TA Entities
NRC on Adoption		National Quality Improvement Center (QIC) Differential Response in Child Protective Services
NRC on Child Protective Services		QIC on Early Childhood
NRC on Child Welfare Data and Technology		QIC on Non-Resident Fathers and the Child Welfare System
NRC on In-Home Services		QIC on Privatization of Child Welfare Services
NRC on Legal and Judicial Issues		QIC on Representation of Children in the Child Welfare System
NRC on Organizational Improvement		National Abandoned Infants Assistance Center
NRC on Permanency and Family Connections		NRC for Community Based Child Abuse Prevention
NRC for Recruitment and Retention of Foster and Adoptive Parents		Child Welfare Information Gateway
NRC for Tribes		National Data Archive on Child Abuse and Neglect
NRC on Youth Development		National Child Welfare Workforce Institute
Implementation Centers		National Center on Substance Abuse and Child Welfare
Atlantic Coast IC		National Technical Assistance Center for Children’s Mental Health
Midwest IC		TA to State Legislators on the Child and Family Services Reviews
Mountains and Plains IC		Technical Assistance Partnership for Child and Family Mental Health
Northeast and Caribbean IC		National Technical Assistance and Evaluation Center for Child Welfare Systems of Care Grantees
Western and Pacific IC		Training and Technical Assistance Coordination Center

2. Do you view these technical assistance providers as part of one overall, integrated network? Please explain your answer.

E3. How did you learn about the T/TA Network and/or any of its specific members?

4. Have any of the resources (e.g., listservs, communities of practice, online brochures or publications, recordings, webpages, links, etc.) provided by the Child Welfare Information Gateway helped to inform you about the available services of members of the T/TA Network?

Yes  Which resources?

No

E.5 Have any of the resources (e.g., T/TA portal, informational brochures, T/TA Request line) of the Children's Bureau's T/TA Coordination Center helped you access or use services from the T/TA Network? 

Yes  Which resources?

No

6. T/TA Engagement and Utilization

Next we have some questions about your involvement with the Children’s Bureau’s Training and Technical Assistance Network.

Interviewer instruction: This information will be “prefilled” in the table(s) for questions F1, F2, and F8 prior to the interview using information obtained from the Children’s Bureau’s technical assistance tracking system.

• If T/TA was received, begin with F1.

• If T/TA was not received, begin with F2.

For each row of the table, read “On [date], [T/TA Provider] provided technical assistance on [topic area] by [mode of TA delivery].”

1. According to the Children’s Bureau’s records of technical assistance activities in the past year,

the [STATE/TRIBE] has received technical assistance from the following IC and NRCs in the following areas:

Date	T/TA Provider	Topic Area	Mode of TA
Prefill	Prefill	Prefill	Prefill

Did we miss anything?

If so, tell us what additional T/TA was received and we will make note of that. Let’s move on now to talk about the technical assistance that was received. SKIP TO F4 

If not, let’s move on now to talk about the technical assistance that was received.

SKIP TO F4 

2. According to the Children’s Bureau’s records of technical assistance activities in the past year, [STATE/TRIBE] has not received any technical assistance from the IC or NRCs.

Is this correct?

Yes, this is correct CONTINUE with F3 

No If the information in OneNet was incomplete or incorrect, continue with the question.

Which IC or NRCs provided technical assistance and what areas were addressed?

Date	T/TA Provider	Topic Area	Mode of TA
Prefill	Prefill	Prefill	Prefill

Let’s move on now to talk about the technical assistance that was received. SKIP TO F4 

Interviewer instruction: For States/Tribes with no T/TA activity, ask F3 and then skip to F8.

3. Did [STATE/TRIBE] request any technical assistance from the National Resource Centers or apply for an Implementation Project through the Implementation Center in the past year?

Yes  Why was technical assistance not provided?

Record response and then SKIP TO F8 

No If T/TA was not requested, continue

Was there a need for technical assistance?

Yes  Why did the state/tribe not pursue technical assistance through the NRCs or IC?

Record response and then SKIP TO F8 

No  SKIP TO F8 

4. As noted in the first question [F1], did the technical assistance provided reflect an understanding of the functioning of the system as a whole? Please explain.

Yes  Can you explain?

No  Why not?

5. Did the technical assistance offer an array of solutions and allow the [STATE/TRIBE] to determine the most appropriate course of action?

Yes  Can you explain?

No  Why not?

5. Did the technical assistance address [STATE’S/TRIBE’S] most pressing issues?

Yes  Can you explain?

No  Why not?

5. Earlier we asked about organizational and system changes that occurred within the child welfare system. Do you believe the T/TA provided by the National Resource Centers or Implementation Centers contributed to this change/these changes?

Yes  For which change(s)?

How were these changes influenced by technical assistance?

No  Why not?

8. Has [STATE/TRIBE] received technical assistance from other T/TA providers in the last year?

Yes  Who provided this T/TA?

What topics or issues were addressed?

Did you pay for these services?

How did you decide which T/TA providers to use?

Was the T/TA effective?

Did the T/TA result in the desired change?

T/TA Provider	Paid Services	Topic Area
Prefill, if available	Prefill, if available	Prefill, if available

No  Why not?

7. Facilitators and Barriers to T/TA Utilization

Interviewer instruction: Ask the following questions of all respondents.

1. There are many factors that may help or hinder [STATE/TRIBE]’s utilization of technical assistance provided by the Implementation Center or National Resource Centers.

Over the last year, were there any key factors that helped [STATE/TRIBE] to use the technical assistance provided by the NRCs or IC?

Yes  What were they?

No

2. From your perspective, over the last year were there any key factors that hindered [STATE/TRIBE] from utilizing the technical assistance provided by the Implementation Center or National Resource Centers?

Yes  What were they?

No

3. To get a complete picture of factors influencing the use of technical assistance at your agency, please tell me if there are any other factors provided on the list below for question G3 that may have helped or hindered the agency’s decision to use TA from the NRCs or IC in the last year.

Additional factors:

Interviewer instruction. For those factors cited, record whether the factor helped or hindered and any comments provided.

Factors

Federal Factors

CFSR findings or development of the PIP

Discussions with the Regional Office

Federal law or policy change

CB T/TA Provider Factors

NRC outreach efforts

IC outreach efforts, such as the regional forums

Participation in the IC’s peer network groups

Participation in the NRCs’ peer network groups

Prior relationship with the NRCs

Prior relationship with the IC

Recommendations from NRCs

Recommendations from IC

T/TA request process

T/TA approval process

Timeliness in which technical assistance can be received

ICs’ level of knowledge and skills

NRCs’ level of knowledge and skills

Geographic proximity of the NRCs

Geographic proximity of the IC in our ACF Region

Cultural competency of the NRCs

Cultural competency of the IC

Usefulness of IC products/materials

Usefulness of NRC products/materials

State/Tribe Factors

State or Tribal incident, such as a child fatality

State’s or Tribe’s financial resources

State’s or Tribe’s availability of staff time

State’s or Tribe’s agency leadership

State’s or Tribe’s attitudes toward making changes to the child welfare system

State’s or Tribe’s attitudes toward seeking outside assistance

External Factors

Informal conversations with colleagues

State/local law or policy change

Lawsuit/legal settlement

Use of technical assistance provided by those outside the CB Network:

• Private consultants

• Other Federal technical assistance providers

• Foundations

• University Partners

H. Implementation Projects

Interviewer instruction: Ask question H1 in Year 2 only. Ask other questions of all respondents in Year 2 and in subsequent years.

1. According to the information we obtained from the Implementation Center semi-annual report, the [STATE/TRIBE]

A1. [Prefill] attended the Regional Forum.

Did your agency find the experience of attending the Regional Forum beneficial?

Yes  Please explain.

No  Please explain.

A2. [Prefill] did not attend the Regional Forum. Can you tell us why you did not attend?

2. According to the information we obtained from the Implementation Center semi-annual report, the [STATE/TRIBE]

A1. [Prefill] did not apply for an Implementation Project.

a. Why did your agency decide not to apply for an Implementation Project?

b. Do you intend to apply in future?

Yes  Please explain. SKIP TO H4 

No  Please explain SKIP TO H4 

A2. [Prefill] applied for an Implementation Project.

a. Proposed Implementation Project [Prefill] was selected.

CONTINUTE TO H3 

b. Proposed Implementation Project [Prefill] was not selected.

1. Did you receive feedback from the [IC NAME] regarding why your project was not selected?

Yes  Was this helpful?

No  Please explain.

2. Were you referred to or did you receive follow-up assistance from other T/TA providers, such as the NRCs or other private providers?

Yes  From whom?

No  Please explain.

3. How are you addressing or how will you address the needs identified in your proposed project?

4. Do you intend to apply again for an Implementation Project?

Yes  Please explain. SKIP TO H4

No  Why not? SKIP TO H4

3. As your agency was selected to have an Implementation Project, we would like to ask you a few questions about the process for establishing the Memorandum of Understanding.

1. To what extent do you feel the STATE/TRIBE was an active participant in developing the terms of the Memorandum of Understanding with the Implementation Center? Please respond using a 5-point rating scale with 1 being not at all active and 5 being very active.

Not at all Active 1

2

3

4

Very Active 5

2. To what extent did the STATE/TRIBE have input into the scope of what was included in the Memorandum of understanding? Please respond using a 5-point rating scale with 1 indicating the State/Tribe had no input and 5 meaning it had a great deal of input.

No Input 1

2

3

4

Great deal of input 5

3. To what extent do you feel the Memorandum of Understanding was developed in a timely manner? Again, respond using a 5-point rating scale with 1 being not at all timely and 5 being extremely timely.

Not at all timely 1

2

3

4

Extremely timely 5

4. To what extent was an effort made to work with the [STATE/TRIBE] in developing a feasible schedule for the work plan. Please respond using a 5-point rating scale with 1 being no effort was made and 5 being a great deal of effort was made.

No effort 1

2

3

4

Great deal of effort 5

Interviewer Instruction: Ask the following questions of all respondents.

3. According to the information we obtained from the Children’s Bureau’s technical assistance tracking system [STATE/TRIBE],

A1. [Prefill] has participated in Peer-to-Peer Networking.

Your State/Tribe participated in the following learning opportunities:

[Read information from below chart]

Did you find the experience beneficial?

Yes  Why?

No  Why not?

T/TA Provider	Peer-to-Peer Event(s)	Topic Area
Prefill	Prefill	Prefill

A2. [Prefill] has not participated in Peer-to-Peer Networking.

Has your State or Tribe participated in any Peer-to-Peer networking events offered by the IC or NRCs, such as participation in Webinars, regional trainings, or peer learning communities?

Yes  Was it helpful?

No  Why not?

5. Do you have any suggestions for improving the peer learning opportunities sponsored by the Children’s Bureau’s T/TA Network?

9. Relationship with T/TA Providers

Now we have a few questions on your opinion of the services provided by the Children’s Bureau’s Training and Technical Assistance providers – that is the NRCs and ICs. Please answer these questions using the following response categories: Not at All, A Little, Somewhat, A Lot, and Very Much So.

Interviewer Instruction: Ask the following questions, depending on the responses to the questions in Section F and/or G.

If T/TA was received from

• Implementation Center only  Ask I1-I6

• National Resource Center only  Ask I7-I12

• both Implementation Center and National Resource Center  Ask I1-I12

If no T/TA was received Ask I1 & I7 only

If respondent responds “not at all,” ”a little,” or “somewhat” to any question, then probe for how that area could be improved.

	Not at all	A little	Somewhat	A lot	Very Much
Implementation Center
How satisfied are you with the level of accessibility of the Implementation Center assigned to your State/Tribe? (Would you say: not at all, a little, somewhat, a lot, or very much so?)
A1. If 1-3: How could accessibility be improved?
When working with your Implementation Center on issues related to the child welfare system in [STATE/TRIBE], how satisfied have you been with the frequency of communication? (Would you say: not at all, a little, somewhat, a lot, or very much so?)
A1. If 1-3: How could this be improved?
How comfortable do you feel disclosing areas of concern or weaknesses of your [STATE/TRIBE] to your Implementation Center? (Would you say: not at all, a little, somewhat, a lot, or very much so?)
A1. If 1-3: How could this be improved?
When working with your Implementation Center, do you feel [STATE/TRIBE] plays an active part in decision-making?
A1. If 1-3: How could this be improved?
How satisfied have you been with the level of follow-through from your Implementation Center?
A1. If 1-3: How could their follow-through be improved?
Overall, how satisfied are you with your relationship with your Implementation Center?
A1. If 1-3: How could your relationship be improved?
National Resource Centers
How satisfied are you with the level of accessibility of the National Resource Centers?
A1. If 1-3: How could accessibility be improved?
When working with National Resource Centers on issues related to the child welfare system in [STATE/TRIBE], how satisfied have you been with the frequency of communication?
A1. If 1-3: How could this be improved?
How comfortable do you feel disclosing areas of concern or weaknesses of your [state/tribe] to National Resource Centers?
A1. If 1-3: How could this be improved?
When working with National Resource Centers, do you feel [STATE/TRIBE] plays an active part in decision-making?
A1. If 1-3: How could this be improved?
How satisfied have you been with the level of follow-through from the National Resource Centers?
A1. If 1-3: How could their follow-through be improved?
Overall, how satisfied are you with your relationship with the National Resource Centers?
A1. If 1-3: How could your relationship be improved?

Interviewer Instruction: Use information from the Children’s Bureau’s technical assistance tracking system to determine if the STATE/TRIBE has participated in a coordinated T/TA effort (e.g., multiple National Resource Center working together, an Implementation Project that uses National Resource Center services).

• If so, ask questions I13-I15.

• If not, skip to Section J.

Now we have a few questions on your experience when working with multiple providers in your [STATE/TRIBE]. Please consider only T/TA provided by multiple National Resource Centers or an Implementation Project that involves the participation of National Resources Centers.

	Not at all	A little	Somewhat	A lot	Very Much
Is there a logical sequence to Training and Technical Assistance when multiple T/TA providers are involved? (Would you say: not at all, a little, somewhat, a lot, or very much so).
If 1-3: How could the sequencing of T/TA be improved?
Are the multiple T/TA providers working in your [STATE/TRIBE] knowledgeable of each others efforts?
If 1-3: How could this be improved?
Overall, how well do the multiple T/TA providers coordinate their activities when they work together in your [STATE/TRIBE]?
If 1-3: How could coordination be improved?

Interviewer Instruction: Ask the following questions of all respondents.

9. Utilization of products and materials from the CB T/TA Network

Now, I would like to ask you about your agency’s use of the Children’s Bureau’s T/TA Network websites, products or materials.

1. Have you personally used any products or materials from the T/TA Network or accessed the websites of its members?

Yes  What did you find to be most useful?

What did you find not useful?

No

2. Do you have any other comments you would like to share regarding the CB T/TA network?

Thank you.

This is the end of the interview.

We greatly appreciate your participation in this evaluation of the Children’s Bureau’s T/TA Network.

Appendix C 20