Instrument 1_SCWS Grant and Partner Survey-8.28.23

OMB Control # xxxx-xxxx and Expiration Date: xx/xx/xxxx

SCWS Evaluation: Grant and Community Partners Survey

Title: Strengthening Child Welfare Systems (SCWS) Evaluation

Protocol No.: None

IRB Protocol #

Sponsor: Children’s Bureau, ACF DHHS

INVESTIGATORS: Elliott Graham, PhD

James Bell Associates

2000 15th Street North, Suite 100
Arlington, VA 22201

United States

Nicole Miller, MSW

James Bell Associates

2000 15th Street North, Suite 100
Arlington, VA 22201

United States

Julie Murphy, MSW

James Bell Associates

2000 15th Street North, Suite 100
Arlington, VA 22201

United States

STUDY-RELATED

PHONE NUMBER(S): Elliott Graham

(703) 842-0958

Nicole Miller

(703) 247-2625

Julie Murphy

(703) 842-0955

Through its Technical Assistance on Evaluation for Discretionary Grant Programs contract with the Children’s Bureau (CB), Administration for Children and Families (ACF), James Bell Associates is tasked with conducting an evaluation of the Strengthening Child Welfare Systems (SCWS) to Achieve Expected Child and Family Outcomes grant cluster. The SCWS evaluation is designed to document the experiences of the five grant recipients around the implementation of their projects and the degree to which they were able to address common child safety, permanency, and well-being outcomes.

The purpose of this survey is to collect information about your experience implementing your SCWS intervention(s) in the last three years of the grant period (Oct 2020-Sept 2023). The information gathered through this survey will provide the CB and the public with an understanding of factors that influenced the implementation of SCWS interventions. The evaluation findings will include successes, challenges, and lessons learned related to the grant interventions and strategies, the contextual factors that affected implementation, and the degree to which grant recipients were able to address child welfare outcomes in their targeted locations.

The survey should take 30 minutes. Your participation in the survey is voluntary. You may refuse to take part in the research or exit the survey at any time without penalty. You may decline to answer any question you do not wish to answer for any reason. There are no foreseeable risks involved in participating in the survey. Your alternative is to not participate. You will receive no direct benefits from participating in this research study. However, your responses will help us learn more about the SCWS discretionary grant cluster and inform the field about effective child welfare interventions.

Your responses to this survey will remain private. Your survey responses will be sent to a link on the Qualtrics survey platform where data will be stored in a password protected electronic format. Only team members overseeing the study will have access to survey records. Survey data will not be associated directly with you but may be associated with the name of the grant and included in a report shared within the Children’s Bureau and ACF, and may also be shared in public reports, articles, and presentations. Information from this study may be securely shared with qualified researchers to help guide future research and support program improvement.

If you have questions or concerns, or complaints about the survey or study, or to report a research-related problem, you may contact Co-Principal Investigators Elliott Graham, Nicole Miller, or Julie Murphy

Co-Principal Investigator

Elliott Graham, PhD

James Bell Associates

2000 15th Street North, Suite 100
Arlington, VA 22201

United States

(703) 842-0958

Co-Principal Investigator

Nicole Miller, MSW

James Bell Associates

2000 15th Street North, Suite 100
Arlington, VA 22201

United States

(703) 247-2625

Co-Principal Investigator

Julie Murphy, MSW

James Bell Associates

2000 15th Street North, Suite 100
Arlington, VA 22201

United States

(703) 842-0955

This research is being overseen by WCG IRB. An IRB is a group of people who perform independent reviews of research studies. You may talk to them at 855-818-2289 or [email protected] if:

• you have questions, concerns, or complaints that are not being answered by the research team.

• you are not getting answers from the research team.

• you cannot reach the research team.

• you want to talk to someone else about the research.

• you have questions about your rights as a research subject.

The Paperwork Reduction Act Statement: The described collection of information is voluntary and will be used to help us gather feedback about the implementation and impact of the SCWS discretionary grant. Public reporting burden for this collection of information is estimated to average 30 minutes, including the time for reviewing instructions, gathering and maintaining the data needed, and participating in the focus group. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number and expiration date for the described collection are OMB #: xxxx-xxxx, Exp: xx/xx/20xx. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to Nicole Miller ([email protected]) and Julie Murphy ([email protected]).

Please select ”agree” or “disagree” below. You may print a copy of this consent form for your records. Clicking on the “Agree” button indicates that:

• You have read the above information.

• You voluntarily agree to participate.

☐ Agree ☐ Disagree

Thank you in advance for your participation and input!

Respondent characteristics

Which state project are you involved in?

• Florida (Embrace Families)

• Kansas (Kansas Strong)

• New Mexico (NMFAP)

• Texas (TXPOP)

• Washington (Permanency From Day 1)

What is your role on the project? [Check all that apply]

• Grant/project staff

• Evaluation staff

• Partner organization

• Other: ____________________ (required)

How long have you been involved in this grant?

• Since the beginning (Sept 2018)

• Since 2019

• Since 2020

• Since 2021

• Since 2022

Level of Collaboration:

1. [Program and evaluation staff only] Select three of your most important partner organizations in this grant; these may be public or private agencies. Without their involvement, successful implementation would not be possible.

   • Public child welfare agency

   • Private child welfare agency

   • Court/legal partner organization

   • Community provider organization

   • Child/family advocacy organization

   • University system (supporting implementation and/or evaluation)

   • Other: Type of organization ________________________________

   • Other: Type of organization ________________________________

(1) Networking: aware of organization, defined roles, little communication, all decisions are made independently

(2) Cooperation: provide information to each other, somewhat defined roles, formal communication, all decisions are made independently

(3) Coordination: share information and resources, defined roles, frequent communication, some shared decision making

(4) Coalition: share ideas, share resources, frequent and prioritized communication, all members have a vote in decision making

(5) Collaboration: members belong to one system, frequent communication is characterized by mutual trust, consensus is reached on all decisions

2. a. [Program and evaluation staff only] We want to understand the level of collaboration between you and your partner agencies prior to being awarded the grant (2018) and at end of the grant period (2023). Using the definitions¹ below, indicate your level of collaboration with each of your three key partner organizations (identified above) in 2018 and 2023.

2018 (1 to 5) 2023 (1 to 5)

Key Partner 1 1 – 2 – 3 – 4 – 5 1 – 2 – 3 – 4 – 5

Key Partner 2 1 – 2 – 3 – 4 – 5 1 – 2 – 3 – 4 – 5

Key Partner 3 1 – 2 – 3 – 4 – 5 1 – 2 – 3 – 4 – 5

2. b. [Partner organizations only] We want to understand the level of collaboration between you and the organization that is the recipient of this grant, prior to being awarded the grant (2018) and at end of the grant period (2023). Using the definitions below, indicate your level of collaboration with the grant recipient in 2018 and 2023.

(1) Networking: aware of organization, defined roles, little communication, all decisions are made independently

(2) Cooperation: provide information to each other, somewhat defined roles, formal communication, all decisions are made independently

(3) Coordination: share information and resources, defined roles, frequent communication, some shared decision making

(4) Coalition: share ideas, share resources, frequent and prioritized communication, all members have a vote in decision making

(5) Collaboration: members belong to one system, frequent communication is characterized by mutual trust, consensus is reached on all decisions

2018 rNetworking rCooperation rCoordination rCoalition rCollaboration

2023 rNetworking rCooperation rCoordination rCoalition rCollaboration

2. Overall, what impact has this grant had on collaboration between you and your partner organizations (e.g., aligned goals and vision, better understanding of diverse perspectives, formal partnership, MOUs, plans for future partnership)? [open ended]

Implementation [GRANT and Evaluation staff only]

1. We are interested in understanding the factors that influenced the successful implementation of your grant interventions. For each factor below, indicate if the factor had any influence on the implementation of your initiatives, the direction of the influence (enhance or hindered implementation) and the level of influence (slight or significant). Note: A single factor may have had both positive and negative influence, so select multiple responses as appropriate.

Factors that influence implementation	No influence on implementation	Enhanced implementation		Hindered implementation		Don’t know	Not applicable
		Sightly	Significantly	Sightly	Significantly
Collaborative partnerships with the child welfare agency
Collaborative partnerships with court
Collaborative partnerships with other partners
Communication strategies about interventions
CQI processes
COVID
Development of Implementation and Evaluation Plan
Organizational readiness
Target population
Availability of Technical Assistance
Workforce challenges (e.g., turnover, lack of qualified candidates)

2. Please elaborate on any of the above topics which were identified as “significantly” enhancing or hindering implementation. [open ended]

3. Are there factors, other than what is noted above, that positively or negatively influenced implementation? If so, describe. [open ended]

IMPACT

1. The goal of this grant is to improve permanency outcomes for child welfare involved families. To what degree do you think the grant will achieve this desired outcome?

r No impact on permanency outcomes

r Some impact on permanency outcomes

r Significant impact on permanency outcomes

r Don’t know

Describe how and why you think this grant will or will not have its intended impact. [open ended]

2. From your perspective, what impact will this grant have on child welfare involved families in your community? [open-ended]

3. [Program and evaluation staff only] The five SCWS grants varied significantly in their approach, targeted audience, interventions, and intended outcomes. The items included in this table are the most common items in the SCWS cluster-level logic model (which was derived from the SCWS funding opportunity announcement and each grant recipient’s logic model).

1. First, indicate if each item is an area of focus for your grant’s interventions/strategies (i.e., an area which your project intended to address or impact.)

2. Second, if it is an area of focus, indicate if you believe your grant interventions will have an impact on the target area.

Area of Focus	Targeted area?	If yes,
		No impact	Some impact	Significant impact
Support child welfare-involved families whose children have not successfully achieved permanency	r Yes r No	r	r	r
Support older youth in care to find permanent homes and reduce time in child welfare custody	r Yes r No	r	r	r
Formalize collaborative partnerships with other child serving organizations	r Yes r No	r	r	r
Enhance partnerships between child welfare and legal/judicial system	r Yes r No	r	r	r
Enhance partnerships with community-based providers	r Yes r No	r	r	r
Improve organizational practices, policies, and procedures	r Yes r No	r	r	r
Improve organizational readiness and capacity	r Yes r No	r	r	r
Build workforce capacity	r Yes r No	r	r	r
Improve organizational functioning of a state/local child welfare agency (e.g., reduced caseloads and staff turnover, improved training programs)	r Yes r No	r	r	r
Utilize findings from CFSR to identify barriers and inform intervention development	r Yes r No	r	r	r
Provide timely and individualized provision of supportive services	r Yes r No	r	r	r
Increase participation in supportive services by children and families	r Yes r No	r	r	r
Enhance case planning and permanency planning policies and processes	r Yes r No	r	r	r
Improve timeliness and efficiency of child welfare and judicial processes and practices	r Yes r No	r	r	r
Involve youth and children in case planning	r Yes r No	r	r	r
Improve communication between child welfare and courts	r Yes r No	r	r	r
Improve collaboration with community-based organizations and service providers	r Yes r No	r	r	r
Increase family engagement	r Yes r No	r	r	r
Impact Permanency Outcome 1: Children have permanency and stability in their living situations	r Yes r No	r	r	r
Impact Permanency Outcome 2: The continuity of family relationships and connections is preserved for children	r Yes r No	r	r	r
Impact Well-Being Outcome 1: Families have enhanced capacity to provide for their children’s needs	r Yes r No	r	r	r
Impact Safety Outcome 1: Children are, first and foremost, protected from abuse and neglect	r Yes r No	r	r	r
Impact Safety Outcome 2: Children are safely maintained in their homes whenever possible and appropriate	r Yes r No	r	r	r
Impact Well-Being Outcome 2: Children receive appropriate services to meet their educational needs	r Yes r No	r	r	r
Impact Well-Being Outcome 3: Children receive adequate services to meet their physical and mental/behavioral health needs	r Yes r No	r	r	r

3. Reflecting on your responses in the above table, identify the top three areas where you expect your grant to have a significant impact and describe each topic’s significant impact in more detail. [open ended]

[open ended]

[open ended]

Other

1. Based on your experience with this grant, what opportunities and lessons did this grant opportunity provide? [open ended]

2. Is there anything else you would like to share regarding the impact of this grant on your local community? [open ended]

1 Note as a reference within Qualtrics. The level of collaboration scale was developed by Frey, Bruce - University of Kansas, 2006.

15