SUPPORTING STATEMENT FOR
Evaluation of the Child Welfare Capacity Building Collaborative
Section B
Submitted by
Department of Health & Human Services
Children’s Bureau
Washington, DC
Contact person:
Brian Deakins
Children’s Bureau
Administration on Children, Youth and Families
Mary E. Switzer Building
3rd Floor, Mailstop 3602
330 C Street, SW
Washington, DC 20201
202-205-8769
Satisfaction Surveys: The target population for the various Satisfaction Surveys consists of child welfare and judicial professionals that use the Centers’ webpages, products, and online courses, that participate in virtual or in-person webinars, meetings, or events, and that receive brief or intensive tailored services from the Centers. Each Satisfaction Survey has different administration procedures for the purposes of the Cross-Center Evaluation. Center Evaluators are free to administer the satisfaction items at additional time points or with a broader set of activities than what is required by the Cross-Center Evaluation.
The Webpages and Products Satisfaction Survey will be administered on Center webpages at the discretion of the Centers and when product content is associated with the priorities set by the Children’s Bureau (e.g., CQI, Preventing Sex Trafficking and Strengthening Families Act) and the Center has a plan to collect outcome data on knowledge gained and attitudes changed, when applicable. The Learning Experience Satisfaction Surveys will be administered by Center Evaluators when the learning experience is part of an ongoing set of classes/instruction that is conducted over three or more sessions, each session/learning activity is one hour in length or longer, and the same participants are targeted to participate in the scheduled sessions, or when the Center has a plan to collect outcome data about the learning experience. The Webinars, Events, and In-Person Meetings Satisfaction Surveys will be administered by Center Evaluators when (1) the peer event is part of an ongoing set of peer networking activities that are held (in-person, by phone, or internet) at least three times per fiscal year with the same constituency group, (2) each peer event session is one hour in length or longer, (3) one of the objectives of the peer event is networking, and (4) the peer event includes three or more representatives from a constituency group; or when the Center has a plan to collect outcome data about the peer event. The Assessment and Capacity Building Satisfaction Survey will be administered by the Cross-Center Evaluation team after a state, tribe, or territory has participated with a Center in assessment and work planning activities.
Center for Tribes Contact Form: The Contact Form will be completed by tribal child welfare program directors or other tribal representatives, or CB Regional Office personnel. This short form briefly outlines a request for services. A tribe can access an electronic version through a website and submit to it the Center. Alternatively, meeting attendees may pick up a paper version, complete, and return the form in-person to a Center for Tribes representative.
Center for Tribes Demographic Survey: The Demographic Survey is completed by the tribal child welfare program director, other individuals in tribal leadership positions, and a Center for Tribes staff person.
Center for Tribes Needs and Fit Exploration Tool Phase 1: A structured telephone interview will be conducted with the tribal child welfare program directors or other tribal representatives to gather more information about the request for services using the NAFET Phase 1 tool.
Center for Tribes Needs and Fit Exploration Tool Phase 2: The NAFET Phase 2 tool will be completed through a structured onsite interview with the tribal child welfare program director and other tribal representatives to gather more information about the tribal child welfare program and their need for capacity building services.
Tribal Organizational Assessment Caseworker Interview: The target population for the Caseworker Interview is tribal child welfare program staff members of tribes that have approved intensive projects or brief tailored services and who provide services to children and families. The Center for Tribes plans to conduct up to 10 brief tailored services projects and 7 intensive tailored services projects per year. The number of caseworkers interviewed for each project is determined on a case-by-case basis, but generally ranges from 2-10 individuals.
Tribal Organizational Assessment Community Provider Interview: The target population for the Community Provider Interview includes representatives of both tribal and non-tribal community-based organizations that provide services accessed by children and families (e.g., substance abuse, mental health, housing, employment) involved with the tribal child welfare program for tribes that have approved intensive projects or brief tailored services. The Center for Tribes plans to conduct up to 10 brief tailored services projects and 7 intensive tailored services projects per year. The number of community providers interviewed for each project is determined on a case-by-case basis.
Tribal Organizational Assessment Community Member/Elder Interview: The target population for the Community Member/Elder Interview includes tribal members and elders who represent the community served by a tribal child welfare program that has an approved intensive project or brief tailored service. The Center for Tribes plans to conduct up to 10 brief tailored services projects and 7 intensive tailored services projects per year. The number of community members/elders interviewed for each project is determined on a case-by-case basis.
Tribal Organizational Assessment Family Interview: The target population for the Family Interview includes families who have received services from a tribal child welfare program that has an approved intensive project or brief tailored service. The Center for Tribes plans to conduct up to 10 brief tailored services projects and 7 intensive tailored services projects per year. The number of families interviewed for each project is determined on a case-by-case basis.
Center for States Information and Referral Survey: This survey will be administered to all State and territory public child welfare agency staff who access information and referral services.
Center for States Intensive Projects Survey: This survey will be administered to all State, territory, and Tribal public child welfare agency staff who participate in intensive “tailored” services engagement.
Center for States Constituency Group Survey: This survey will be administered to all State and territory public child welfare agency staff who participate in constituency groups.
Center for States Brief Tailored Services Survey: This survey will be administered to all State, territory, and Tribal public child welfare staff who participate in Brief Services.
CIP Annual Meeting Survey: The target population for the CIP Annual Meeting Survey is all attendees at the meeting; approximately 200 per year. This includes Court Improvement Program staff, collaborative partners, and Center for Courts support staff persons who are present. All attendees must register for the event. The registration data is used to identify all persons within the target population. Center for Courts evaluation staff administers the survey via an electronic link within two weeks of the event. A follow-up reminder is sent one week later to encourage participation.
Center for Courts CQI Workshops: The target population for the Center for Court CQI Workshops survey is all participants at the workshop. Workshop participants will vary depending on the focus of the workshop and the needs of the CIP, but typically includes Court Improvement Program staff, collaborative partners, and consultants who may be working on specific projects. Approximately 16 stakeholders participate in each workshop. The workshop survey is administered by hand at the end of workshop by Center for Courts staff.
Leadership Interviews: The potential respondent universe for the Leadership Interviews includes Child Welfare Directors (or their designee) from all 56 States and Territories; Directors of Court Improvement Programs (CIPs) of CIPs that receive services from the Center for Courts (up to 52 of the 52 State CIPs); and Child Welfare Directors of Tribes that receive services from the Center for Tribes (estimated number 30) Two samples will be drawn. A census sample of Child Welfare Directors (or their designees) from the 56 States and Territories will be included. This sample comprises all agencies that are federally monitored through the Child and Family Services Review and entitled to receive capacity building services from the CB providers. A census is necessary to obtain all 56 child welfare leaders’ perceptions and perspectives about service utilization and services received with regard to specific change initiatives in their respective child welfare systems.
A census sample will also be drawn of the CIP and Tribal Child Welfare Directors (or their designees). This sample comprises all CIPs and Tribal agencies that receive services from the Capacity Building Centers. Up to 52 CIPs and an estimated 30 Tribal Child Welfare agencies will be invited to participate in the Leadership Interview.
An estimated total of 134 State, CIP, and Tribal Directors will be selected for survey administration. The baseline survey will be administered in FY 2016. Follow-up survey will be administered in FY 2018. There has been no previous collection.
Collaboration Survey: The potential respondents for the Collaboration Survey are the approximately 230 individuals who serve as staff members and consultants of the three centers that comprise the Children’s Bureau Capacity Building Collaborative. The expected response rate is 70 percent. The baseline survey will be administered in FY 2016, and follow-up surveys will be administered in FY 2017 and 2018. There has been no previous collection.
No statistical methodology for stratification and sample selection will be used for any of the data collection instruments.
Maximizing response rates is critical to the administration of the aforementioned surveys.1 The content and format of the instruments were developed in close consultation with key stakeholders, including the collaborative centers and were informed by previously developed measures involving technical assistance provision.
Strategies that emphasize flexibility, confidentiality, and a respect for the respondent’s time facilitate timely participation. The following strategies will be implemented to maximize participation in the data collection and achieve the desired response rates: 2
Introduction and notification: Strategies to introduce and notify respondents about data collection are used for several instruments. An introductory letter will be sent on CB letterhead to inform all State, CIP, and Tribal respondents about the administration of the Leadership Interviews. A description of the cross-center evaluation will be included in this mailing. For prospective Tribal respondents, this letter of introduction will also be sent to the Tribal Leader or Chairperson. Follow-up introductory calls will be made with prospective respondents to introduce the evaluation team and to address any questions about the data collection. In the subsequent years, reminder emails will be sent or telephone calls will be made to all State, CIP, and Tribal respondents. We recognize that there may be some turnover in leadership over time and that we may have to re-introduce the survey. The CB will notify the three center directors about the administration of the Collaboration Survey through a listserv announcement and annual meetings of this group. For the CQI Workshop survey, participants are told at the beginning of the workshop that their feedback is critical to helping us improve programs and are provided a survey at the end of the workshop. For the Center for States, all recipients of brief and intensive services, learning experiences, peer networking events, and constituency groups, will be notified at the beginning of service delivery of the Center’s evaluation efforts and the desire for their participation.
Timing of data collection: Discussions were held with stakeholders to determine optimal periods for data collection in order to minimize respondent burden and to facilitate recall. The Leadership Interviews will be conducted during the months of May-June in FYs 2016 and 2018. Administration will be coordinated with data collections efforts conducted by the Centers. The various Satisfaction Surveys will be administered throughout the project period following the administration guidelines described in section B-1. In order to maximize recall the Collaboration Survey will be administered annually every June, in FYs 2016, 2017, and 2018. The Center for Tribes Contact Form and NAFET I tool will be completed when a project is requested by a tribe. The Demographic Survey and NAFET 2 tool will be administered following project approvals. Center for Tribes Tribal Organizational Assessment interviews will be scheduled following approval of all intensive tailored services projects and applicable brief tailored services projects.
Pre-interview preparation: A copy of the Leadership Interview will be sent to respondents in advance of the telephone interview. Background information for certain survey items will be “pre-filled” using information obtained from semi-annual reports or agency websites. Prior interviewer knowledge or familiarity with each State or Tribe’s child welfare system or Court Improvement Program will expedite administration of the interview. Pre-interview preparation is not applicable to the Satisfaction Surveys and the Collaboration Survey. The tribal child welfare program director will be provided a copy of the Center for Tribes Demographic Survey in advance of survey completion. No pre-interview preparation is needed for the NAFET 1 and 2 Tool or the Tribal Organizational Assessment interviews.
Administration: For the Satisfaction Surveys, an email notification that will be sent to the appropriate participants with a request to complete the survey (i.e., by accessing a web-link to an online version of the survey or accessing an attached survey to complete and return via email, mail or secure fax). Weekly reminder emails will be sent for two weeks to respondents who have not yet completed the surveys. Tribal representatives or CB Regional Office personnel will initiate data collection for the Center for Tribes Contact Form and NAFET 1 Tool. A Center for Tribes staff person and the tribal child welfare program director will mutually determine the time for completing the Center for Tribes Demographic Survey and the NAFET 2. Likewise, Center for Tribes staff and the tribal child welfare program director will set dates for on-site interviewing for the Tribal Organizational Assessment. Interviewees will select an interview time during which they are available during those dates. For the CQI Workshop survey, surveys will be administered by hand at the conclusion of the event. Participants will be provided the survey and asked to turn in to a designated folder/location to ensure anonymity of responses. For the CIP Annual Meeting survey, all registered participants will receive an email from the Center for Courts evaluation team asking them to please participate in the survey. They will be provided an email link to take the survey electronically. A follow-up email reminder will be sent out after one week to increase response rates. For the Leadership Interviews, the telephone interviews will be scheduled at the respondents’ convenience. The evaluation team will confirm the interview 2-3 days beforehand and re-schedule any interviews as necessary to accommodate any changes in a Director’s schedule, given the dynamic work environment of public child welfare agencies.3 Similarly, the evaluation team will do the same with Tribal and CIP respondents, given the potential for schedule changes due to community obligations or seasonal fluctuations in cultural activities.4 For the Collaboration Surveys, an email notification will be sent to all center staff and consultants with a request to complete the survey (i.e., by accessing a web-link to an online version of the survey or accessing an attached survey to complete and return via email, mail or secure fax). Electronic participation will allow respondents the flexibility to complete the survey at the most convenient time with minimal burden. For approximately four weeks after sending this initial email, weekly reminder emails will be sent to those respondents who have not yet completed the survey, along with updates on response rates to Center directors.
Alternate response methods: Respondents will be given the option to use an alternate method for the Leadership Interviews. For the Leadership Interview if a respondent prefers to submit written responses to a survey in lieu of participating in a telephone interview, we will provide him/her with a paper version to complete via fax, email, or mail. Similarly, paper versions of any of the electronic surveys will be sent to respondents upon request or will be administered through a telephone interview if requested to accommodate any special needs.
Assurances of data confidentiality: Respondents to all surveys will be assured that reported data are aggregated and not attributable to individuals or organizational entities.
There are no incentives provided for participation in any of the surveys.
The Cross-Center evaluation instruments contained herein were subject to review and feedback by key stakeholders, including the CB, and staff from all three Centers. The Center for Courts’ instruments were reviewed by key stakeholders, the CB, and Center staff.
Each of the instruments was pilot tested to confirm survey item validity and to identify possible procedural or methodological challenges in need of attention or improvement. Pilot tests were conducted for each instrument using a sample of no more than nine respondents (i.e., former State and Tribal child welfare Directors, former CIP directors, former child welfare agency personnel, and current Center staff). Following the pilot tests, the instruments were refined to minimize burden and improve utility. The pilot tests were instrumental in determining the amount of time required to complete the surveys and forms and develop the burden estimates.
User access and responsiveness to the web-based methodology for completing the Satisfaction Survey and Collaboration Survey was also pilot-tested. No modifications will be needed for implementation of the full data collection.
National Cross-Center Evaluation Contractor |
|
James Bell Associates 1001 19th Street North, Suite 1500 Arlington, VA (703) 528-3230 |
ICF International 10503 Rosehaven Street Suite 400 Fairfax, VA 22030 |
National Capacity Building Center for States |
|
ICF International 10503 Rosehaven Street Suite 400 Fairfax, VA 22030 |
|
National Capacity Building Center for Tribes |
|
Butler Institute for Families University of Denver, Graduate School of Social Work 2148 S. High St. Denver, CO 80208 303/871-4548 |
|
National Capacity Building Center for Courts |
|
National Council of Juvenile and Family Court Judges PO Box 8970 Reno, NV 89507 (775)-784-4883 |
|
Cross-Site and Center Evaluation Research Questions
Cross-Center Evaluation Questions
How and to what extent are key partners across and within the Centers collaborating?
What are the capacity building service interventions? To what degree are they evaluable? (Have they been sufficiently defined? Do we know what the defining characteristics, critical elements of the interventions are? Can we assess the degree to which these interventions are being delivered as intended and received? What assistance needs to be provided to help further specify these interventions?)
Universal service/dissemination strategies
Constituency-focused services
Tailored services
To what degree are Centers following common protocols for:
Product design?
Selecting dissemination strategies?
Learning experience choices and development?
Assessment and work planning?
Designing Intensive projects?
To what degree are service interventions being delivered/performed as designed? (What is being received? (i.e., intensity, frequency, etc.)
Dissemination strategies
Learning experiences
Peer networking activities
Assessment and work planning
Brief services
Intensive projects
How satisfied are recipients with the services received?
Universal services
Website content
Products and dissemination strategies (that we may be able to further categorize)
Constituency-focused services
Learning experiences
Peer networking activities
Tailored services
Assessment & work planning
Brief tailored services
Intensive projects (12-month work plans, MOUs, milestone schedules, etc.)
How effective were the service interventions?
Dissemination strategies
Learning experiences
Peer networking activities
Brief tailored services
Intensive tailored services
Which service approaches were most effective and under what conditions? (How did other factors influence the degree to which services were effective? Can this inform our decisions about the service strategies we choose in the future?)
Dissemination strategies
Learning experiences
Peer networking activities
Brief tailored services
Intensive tailored services
What are the costs of Universal, Constituent-focused, and Tailored products, activities, and services?
The data collection plan to address each of the evaluation questions is discussed in the sections below.
How do Centers operationalize the CB evidence-informed steps? What practices have the Centers identified as key to helping States to be successful in implementing these steps? (i.e. what is the Centers’ understanding of the steps? How consistent are they in applying the evidence-informed steps in their work?)
Capacity Building Center for States’ Evaluation Questions
Are project activities progressing on time and within expectations?
What are the services provided by the Center? Are deliverables being met?
What are Center staff perceptions regarding project structures, processes, and collaboration and their impact on the effectiveness of service provision? How do they evolve over the life of the Center?
To what extent are universal, constituency, and tailored services implemented with fidelity to coordinated and established processes? How do variations in processes and procedures impact service quality and effectiveness?
What are the structures in place to support continuous quality improvement? How does review of data impact service improvement and modification (e.g., changing work plans)?
As services become more solidified and refined, what are the lessons learned from Center development that can be shared with the field?
How do recipients perceive services provided by the Center?
Are Center services timely, accessible and in the right format for meeting needs?
Are Center services considered useful and relevant?
Are Subject Matter Experts of high quality and appropriate for assigned tasks?
How does State agency perception of engagement in the process of assessment, planning, and service delivery relate to the perceptions of service quality and effectiveness?
How and to what extent does each type of service (Universal, Constituency, Tailored) increase capacity?
Universal and Constituency:
How are priorities for universal and constituency services defined and determined?
What short-term outcomes in knowledge, skills, and attitudes result from universal and constituency services?
What changes in policy, practice, programs, and processes result from constituency services?
How are collaborative constituency services (e.g., peer networking) improving relationships across agencies and increasing knowledge, capacity, and efficient use of resources?
Tailored Services:
Does the collaborative and evidence-based assessment process lead to informed, realistic, and high quality work plans?
What short-term outcomes in knowledge, skills, and attitudes result from tailored services?
Which tailored services strategies were associated with achieving work plan goals?
What contextual factors influence successful accomplishment of work plans goals?
What changes in policy, practice, programs, and processes result from tailored services?
Capacity Building Center for Tribes’ (CBCT) Evaluation Questions
What were the strengths and challenges that affected the implementation of the CBCT?
How successful has the CBCT been in sustaining a collaborative governance structure that supports effective provision of capacity building services for tribal child welfare programs?
Universal Capacity Building Services
How successful is the CBCT in reaching a broad range of child welfare stakeholders with resources, information, and tools for serving AI/AN families?
Which stakeholders utilize the CBCT tools and resources, and how do these stakeholders perceive the accessibility, relevance, and usability of these resources?
What elements support child welfare stakeholders’ access to and utilization of information, tools, and resources related to serving AI/AN families? What challenges or barriers exist to access and utilization among these stakeholders?
Targeted Capacity Building Services
In what ways have the CBCT resources, strategies, and Targeted Capacity Building Services increased the knowledge, skills, and application of these interventions for recipients of tribal capacity building services?
How effective are P2P networks in fostering learning, support, collaboration, and innovation?
To what degree does the blended learning professional development series training advance the preparation of a culturally responsive and skilled child welfare workforce?
Tailored Capacity Building Services
In what ways have Tailored Capacity Building Services improved tribal child welfare program administration and practice?
How have Tailored Capacity Building Services improved tribal child welfare organizational processes, policies, and infrastructure?
How have Tailored Capacity Building Services helped to create a positive work environment and address workforce issues?
How have Tailored Capacity Building Services improved tribal child welfare programs’ interagency and cross-system collaboration?
Permanency Projects
In what ways have Permanency Projects built capacity in tribal child welfare programs to improve permanency outcomes for children in out-of-home placement?
What strategies or programs have tribes implemented to address the needs of children and families in adoption and guardianship situations?
Title IV-E Tribes
How has support for title IV-E tribes helped improve tribal–state agreements/collaboration?
In what ways have CBCT supports assisted tribes to build capacity to apply for a planning grant? To prepare a direct title IV-E application? To develop and implement a title IV-E program?
What supports do tribes need to successfully implement a direct title IV-E program?
Capacity Building Center for Courts’ (CBCC) Evaluation Questions
What services does CBCC provide? (Process Measures)
How many interactions (e.g., consultation, coaching, training, strategic planning, CQI efforts) do CBCC staff have with CIPS?
How many universal capacity building resources developed and distributed?
How many resources are accessed or downloaded online?
How many CIPs are pursuing development of evidence-informed/based practices?
How many subject specific materials are developed and disseminated?
How many targeted TA resources are provided around a specific topic?
How many locations are served with on-site targeted technical assistance?
How many regional and national events are held?
How many peer-to-peer learning and sharing opportunities are facilitated?
How many CIP collaborative events are held?
How many online training opportunities are developed and disseminated?
How many hours of TA are provided in collaboration with other T/TA capacity building providers?
How do CIPs perceive the services provided by CBCC? (Process/Satisfaction)
Do CIPs find the online resources easy to navigate?
Are tools developed by CBCC used by CIPs? In what way?
Are services provided timely?
Are tools, resources, and services of high quality and useful to CIPs?
Do CBCC staff and consultants have the appropriate skills and knowledge to assist the CIPs?
Do CIPs find CBCC staff helpful? In what ways?
How likely are CIPs to change or implement a new practice as a result of services provided by CBCC?
How does the focus on continuous quality improvement help CIPs to better work with their courts?
Are CBCC efforts related to specific changes in knowledge, behavior, or practice, including: (Outcome)
Increasing the CIPs understanding and application of continuous quality improvement to projects and activities;
Improving the CIPs ability to conduct ongoing assessments;
Improving the CIPs ability to collect, analyze and use data;
Increasing the CIPs collaboration with Agencies, Tribes, and stakeholders;
Increasing the CIPs meaningful participation in federal monitoring processes;
Increasing the number of shared goals and strategies between CIPs and CW agencies;
Increasing the capacity of CIPs to meaningfully participate in joint CQI efforts with the child welfare agency;
Increasing CIPs awareness and understanding of research on best practices;
Improving the CIPs ability to share data and supporting interoperability?
How effective are specific CBCC efforts in terms of building capacity? (Outcome)
Do CIPs report and increased understanding of how to improve or assess practice related to a specific topic area following participation in a constituency group?
Do CIPs report increases in collaborative practices following participation in a constituency group?
Do CIPs report a better understanding of continuous quality improvement in general or specific to a topic of interest after accessing resources available on the CBCC website?
Do CIPs initiate more robust assessments, evaluations, or research related to specific practices following engaging in specific CBCC activities (e.g., CQI workshops, CQI consults)?
Do CIPs demonstrate an increased awareness of resources and research after visiting the CBCC website?
Do CIPs show an increased interest in and understanding of federal monitoring?
Do CIPs demonstrate increased knowledge related to specific topical areas after viewing online training materials?
Do CIPs demonstrate increased knowledge related to specific topical areas after participation in a CQI workshop focused on that topical area?
How is CBCC contributing to the identification and dissemination of empirically supported best practices in dependency court?
Are CBCC efforts increasing the capacity of CIPs to improve safety, permanency, and well-being for children in foster care?
2 Strategies that pertain to two or more data collections are discussed together.
3 Brooks, D. & Wind, L.H. (2002). Challenges implementing and evaluating child welfare demonstration projects. Children and Youth Services Review, 24, nos. 6/7, 379-383. Solomon, B. (2002). Accountability in public child welfare: Linking program theory, program specification and program evaluation. Children and Youth Services Review, 24, nos. 6/7, 385-407.
4 Running Wolf, P., Soler, R., Manteuffel, B., Sondheimer, D., Santiago, R.L., Erickson, J. (2004). Cultural Competence Approaches to Evaluation in Tribal Communities. Rockville, MD: Substance Abuse and Mental Health Services Administration.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Jan Rothstein |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |