Evaluation of the Child Welfare Capacity Building Collaborative: Part One
Supporting Statement Part B
Department of Health and Human Services
Children’s Bureau
Washington, DC
Contact person:
Brian Deakins
Children’s Bureau
Administration on Children, Youth and Families
Mary E. Switzer Building
3rd Floor, Mailstop 3602
330 C Street, SW
Washington, DC 20201
202-205-8769
Section B: Collections of Information Employing Statistical Methods
The Evaluation of the Child Welfare Capacity Building Collaborative: Part One is sponsored by the Children’s Bureau (CB) in the Administration on Children, Youth and Families (ACYF), Administration for Children and Families (ACF), U.S. Department of Health and Human Services (DHHS) which seeks approval for the data collection instruments and procedures described herein. The proposed information collection is necessary to continue to facilitate, track, monitor, and evaluate the activities of the Capacity Building Collaborative which includes three federally funded centers (Center for States, Center for Tribes, and Center for Courts) that deliver national child welfare expertise and evidence-informed training and technical assistance services to State, Tribal, and Territorial public child welfare agencies and Court Improvement Programs (CIPs).
The CB-funded Centers’ collective goal is to build the capacities of State, local, and Tribal child welfare systems to successfully undertake practice, organizational, and systemic reforms necessary to implement federal policies, meet federal standards, and achieve better outcomes for the children, youth and families they serve.
This information collection request is for continued approval of instruments previously approved for the first part of data collection being conducted for the evaluation of the Child Welfare Capacity Building Collaborative. This first group of instruments was submitted and approved under OMB #0970-0484 in August 2016. Approval for continued use through 2022 is being sought for all but four of the originally submitted instruments ((Tribal Organizational Assessment Interviews: Caseworker Interview; Community Provider Interview; Community Member/Elder Interview; Family Interview). This supporting statement details the first phase of data collection as part of this evaluation.
Satisfaction Surveys: The target population for the various Satisfaction Surveys consists of child welfare and judicial professionals that use the Centers’ webpages, products, and online courses, that participate in virtual or in-person webinars, meetings, or events, and that receive brief or intensive tailored services from the Centers. Each Satisfaction Survey has different administration procedures for the purposes of the Cross-Center evaluation. Center evaluators are free to administer the satisfaction items at additional time points or with a broader set of activities than what is required by the Cross-Center evaluation.
The Webpages and Products Satisfaction Survey will be administered on Center webpages at the discretion of the Centers and when product content is associated with the priorities set by the Children’s Bureau and the Center has a plan to collect outcome data on knowledge gained and attitudes changed, when applicable. The Learning Experience Satisfaction Surveys will be administered by Center evaluators when the learning experience is part of an ongoing set of classes/instruction that is conducted over three or more sessions, each session/learning activity is one hour in length or longer, and the same participants are targeted to participate in the scheduled sessions, or when the Center has a plan to collect outcome data about the learning experience. The Webinars, Events, and In-Person Meetings Satisfaction Surveys will be administered by Center evaluators when (1) the peer event is part of an ongoing set of peer networking activities that are held (in-person, by phone, or web-based) at least three times per fiscal year with the same constituency group, (2) each peer event session is one hour in length or longer, (3) one of the objectives of the peer event is networking, and (4) the peer event includes three or more representatives from a constituency group; or when the Center has a plan to collect outcome data about the peer event. The Assessment and Capacity Building Satisfaction Survey will be administered by the Cross-Center evaluation team after a state, tribe, territory, or CIP has participated with a Center in assessment and work planning activities.
Center for Tribes Contact Form: The Contact Form will be completed by tribal child welfare program directors or other tribal representatives, or CB Regional Office personnel. This short form briefly outlines a request for services. A tribe can access an electronic version through a website and submit it to the Center. Alternatively, meeting attendees may pick up a paper version, complete, and return the form in-person to a representative of the Center for Tribes.
Center for Tribes Demographic Survey: The Demographic Survey is completed by the tribal child welfare program director, other individuals in tribal leadership positions, and a Center for Tribes staff person.
Center for Tribes Needs and Fit Exploration Tool - Phase 1 (NAFET 1): The target population for the Center for Tribes Needs and Fit Exploration Tool-Phase 1 is tribes that make a request to the Center for Tribes for services. When a tribe contacts the Center, a Center staff person will hold a telephone discussion with 1-2 representatives using the semi-structured questions and discussion topic areas in the NAFET 1 tool to help these individuals describe their program and the issues that have led them to seek assistance. The NAFET 1 tools help the Center for Tribes get an initial impression of whether the tribe’s request for assistance aligns with the services offered by the Center.
Center for Tribes Needs and Fit Exploration Tool - Phase 2 (NAFET 2): The target population for the Center for Tribes Needs and Fit Exploration Tool - Phase 2 is tribes that participated in the NAFET 1, whose request is a fit for Center services, and who are interested in moving forward with designing a capacity-building project. The semi-structured questions and discussion topic areas in the NAFET 2 tool will be administered during the Center’s first on-site visit with a group of tribal representatives. The information gathered from both the NAFET 1 and NAFET 2 tools will be key sources of assessment data that will be used by the Center’s site lead to understand the tribe’s capacity and readiness to implement its project.
Center for States Information and Referral Survey: This survey will be administered to all State and territory public child welfare agency staff who access information and referral services.
Center for States Intensive Projects Survey: This survey will be administered to all State, territory, and Tribal public child welfare agency staff who participate in intensive “tailored” services engagement.
Center for States Constituency Group Survey: This survey will be administered to all State and territory public child welfare agency staff who participate in constituency groups.
Center for States Brief Tailored Services Survey: This survey will be administered to all State, territory, and Tribal public child welfare staff who participate in Brief Services.
CIP Annual Meeting Survey: The target population for the CIP Annual Meeting Survey is all attendees at the meeting; approximately 200 per year. This includes CIP staff, collaborative partners, and Center for Courts support staff persons who are present. All attendees must register for the event. The registration data is used to identify all persons within the target population. Center for Courts evaluation staff administers the survey via an electronic link within two weeks of the event. A follow-up reminder is sent one week later to encourage participation.
Center for Courts CQI Workshops: The target population for the Center for Court CQI Workshops survey is all participants at the workshop. Workshop participants will vary depending on the focus of the workshop and the needs of the CIP, but typically includes CIP staff, collaborative partners, and consultants who may be working on specific projects. Approximately 16 stakeholders participate in each workshop. The workshop survey is administered by hand at the end of workshop by Center for Courts staff.
Leadership Interviews: The potential respondent universe for the Leadership Interviews includes Child Welfare Directors (or their designee) from all 56 States and Territories; Directors of CIPs that receive services from the Center for Courts (up to 52 of the 52 State CIPs); and Child Welfare Directors of Tribes that receive services from the Center for Tribes (estimated number 39). Two samples will be drawn. A census sample of Child Welfare Directors (or their designees) from the 56 States and Territories will be included. This sample comprises all agencies that are federally monitored through the Child and Family Services Review and entitled to receive capacity building services from the CB providers. A census is necessary to obtain all 56 child welfare leaders’ perceptions and perspectives about service utilization and services received about specific change initiatives in their respective child welfare systems.
A census sample will also be drawn of the CIP and Tribal Child Welfare Directors (or their designees). This sample comprises all CIPs and Tribal agencies that receive services from the Capacity Building Centers. Up to 52 CIPs and an estimated 39 Tribal Child Welfare agencies will be invited to participate in the Leadership Interview.
An estimated total of 147 State, CIP, and Tribal Directors will be selected for interviews. The baseline interview will be conducted in FY 2020. Follow-up interviews will be administered in FY 2022.
Collaboration Survey: The potential respondents for the Collaboration Survey are the approximately 230 individuals who serve as staff members and consultants of the three centers that comprise the Children’s Bureau Capacity Building Collaborative. The expected response rate is over 70 percent. The baseline survey will be administered in FY 2020, and follow-up surveys will be administered in FY 2021 and 2022.
No statistical methodology for stratification and sample selection will be used for any of the data collection instruments.
An introductory letter will be sent on CB letterhead to inform all State, CIP, and Tribal respondents about the administration of the Leadership Interviews. The introductory letter includes a brief description of the Cross-Center evaluation. For prospective Tribal respondents, this letter of introduction will also be sent to the Tribal Leader or Chairperson. Follow-up introductory letters and phone calls will be made with prospective respondents to introduce the evaluation team and to address any questions about the data collection. In the subsequent years, reminder emails will be sent, or telephone calls will be made to all State, CIP, and Tribal respondents. (see Appendix E for copies of introductory materials with interview participants). We recognize that there may be some turnover in leadership over time and that we may have to re-introduce the survey. The Leadership Interviews will be conducted in FYs 2020 and 2022. Administration will be coordinated with data collections efforts conducted by the Centers.
The Leadership Interviews will be scheduled at the respondents’ convenience. The evaluation team will confirm the interview 2-3 days beforehand and re-schedule any interviews as necessary to accommodate any changes in a Director’s schedule, given the dynamic work environment of public child welfare agencies.1 Similarly, the evaluation team will do the same with Tribal and CIP respondents, given the potential for schedule changes due to community obligations or seasonal fluctuations in cultural activities.2 For the Leadership Interview if a respondent prefers to submit written responses to a survey in lieu of participating in a telephone interview, we will provide him/her with a paper version to complete via fax, email, or mail. Similarly, paper versions of any of the electronic surveys will be sent to respondents upon request or will be administered through a telephone interview if requested to accommodate any special needs.
The CB will notify the three center directors about the administration of the Collaboration Survey during one of its regular meetings with this group. An email notification will be sent to all center staff and consultants with a request to complete the survey (i.e., by accessing a web-link to an online version of the survey or accessing an attached survey to complete and return via email, mail or secure fax). Electronic participation will allow respondents the flexibility to complete the survey at the most convenient time with minimal burden. For approximately four weeks after sending this initial email, weekly reminder emails will be sent to those respondents who have not yet completed the survey, along with updates on response rates to Center directors (see Appendix E for copies of the survey introductions and reminder emails to be sent to survey participants). In order to maximize recall, the Collaboration Survey will be administered annually in FYs 2020, 2021, and 2022.
For the CQI Workshop survey, participants are told at the beginning of the workshop that their feedback is critical to helping improve programs. Surveys will be administered by hand at the conclusion of the event. Participants will be provided the survey and asked to turn it in to a designated folder/location to ensure anonymity of responses. For the CIP Annual Meeting survey, all registered participants will receive an email from the Center for Courts evaluation team asking them to please participate in the survey. They will be provided an email link to take the survey electronically. A follow-up email reminder will be sent out after one week to increase response rates.
For the Center for States, all recipients of brief and intensive services, learning experiences, peer networking events, and constituency groups, will be notified at the beginning of service delivery of the Center’s evaluation efforts and the desire for their participation.
Tribal representatives or CB Regional Office personnel will initiate data collection for the Center for Tribes Contact Form. The Center for Tribes Demographic Survey will be administered following project approvals. A Center for Tribes staff person and the tribal child welfare program director will mutually determine the time for completing the Center for Tribes Demographic Survey. The Center for Tribes NAFET 1 and NAFET 2 tools will be employed when tribes initiate a request for services (NAFET 1) and when tribes opt to engage in a brief or intensive project (NAFET 2).
The various Satisfaction Surveys (e.g., Web Pages and Products Satisfaction Survey; Learning Experiences Satisfaction Surveys; Webinars, Events, and In-Person Meetings Satisfaction Surveys; Assessment and Capacity Building Work Plan Satisfaction Survey) will be administered throughout the project period following the administration guidelines described in section B-1. For the Satisfaction Surveys, an email notification will be sent to the appropriate participants with a request to complete the survey (i.e., by accessing a web-link to an online version of the survey or accessing an attached survey to complete and return via email, mail or secure fax). Weekly reminder emails will be sent for two weeks to respondents who have not yet completed the surveys (see Appendix E for copies of the survey introductions and reminder emails to be sent to survey participants).
There are no incentives provided for participation in any of the surveys.
Maximizing response rates is critical to the administration of the aforementioned surveys.3 The content and format of the instruments were developed in close consultation with key stakeholders, including the collaborative Centers and were informed by previously developed measures involving technical assistance provision.
Strategies that emphasize flexibility, privacy, and a respect for the respondent’s time facilitate timely participation. The following strategies will be implemented to maximize participation in the data collection and achieve the desired response rates: 4
Introduction and notification: Strategies to introduce and notify respondents about data collection are used for several instruments.
Timing of data collection: Discussions were held with stakeholders to determine optimal periods for data collection in order to minimize respondent burden and to facilitate recall. Data collection will be scheduled to align with these optimal periods.
Pre-interview preparation: A copy of the Leadership Interview will be sent to respondents in advance of the telephone interview. Background information for certain survey items will be “pre-filled” using information obtained from semi-annual reports or agency websites. Prior interviewer knowledge or familiarity with each State or Tribe’s child welfare system or CIP will expedite administration of the interview. Pre-interview preparation is not applicable to the Satisfaction Surveys and the Collaboration Survey. The tribal child welfare program director will be provided a copy of the Center for Tribes Demographic Survey in advance of survey completion.
Administration: Reminder emails will be sent (per discussion above) to promote participation and a high response. For interviews, staff will be flexible and accommodating with respect to rescheduling as necessary and setting up interviews to meet participants’ availability.
Alternate response methods: Respondents will be given the option to use an alternate method for the Leadership Interview, such as submitting a paper version with written responses to questions, if this method helps to increase participation.
Assurances of data privacy: Respondents to all surveys will be assured that reported data are aggregated and not attributable to individuals or organizational entities.
The Cross-Center evaluation instruments contained herein were subject to review and feedback by key stakeholders, including the CB, and staff from all three Centers. The Center-specific instruments were reviewed by key stakeholders, the CB, and Center staff.
Each of the instruments was pilot tested to confirm survey item validity and to identify possible procedural or methodological challenges in need of attention or improvement. Pilot tests were conducted for each instrument using a sample of no more than nine respondents (i.e., former State and Tribal child welfare Directors, former CIP directors, former child welfare agency personnel, and current Center staff). Following the pilot tests, the instruments were refined to minimize burden and improve utility. The pilot tests were instrumental in determining the amount of time required to complete the surveys and forms and develop the burden estimates.
User access and responsiveness to the web-based methodology for completing the Satisfaction Survey and Collaboration Survey was also pilot-tested. No modifications will be needed for implementation of the full data collection.
National Cross-Center Evaluation Contractor |
|
James Bell Associates 1001 19th Street North, Suite 1500 Arlington, VA (703) 528-3230 |
ICF International 10503 Rosehaven Street Suite 400 Fairfax, VA 22030 |
National Capacity Building Center for States |
|
ICF International 10503 Rosehaven Street Suite 400 Fairfax, VA 22030 |
|
National Capacity Building Center for Tribes |
|
Butler Institute for Families University of Denver, Graduate School of Social Work 2148 S. High St. Denver, CO 80208 303/871-4548 |
|
National Capacity Building Center for Courts |
|
National Council of Juvenile and Family Court Judges PO Box 8970 Reno, NV 89507 (775)-784-4883 |
|
1 Brooks, D. & Wind, L.H. (2002). Challenges implementing and evaluating child welfare demonstration projects. Children and Youth Services Review, 24, nos. 6/7, 379-383. Solomon, B. (2002). Accountability in public child welfare: Linking program theory, program specification and program evaluation. Children and Youth Services Review, 24, nos. 6/7, 385-407.
2 Running Wolf, P., Soler, R., Manteuffel, B., Sondheimer, D., Santiago, R.L., Erickson, J. (2004). Cultural Competence Approaches to Evaluation in Tribal Communities. Rockville, MD: Substance Abuse and Mental Health Services Administration.
4 Strategies that pertain to two or more data collections are discussed together.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | James DeSantis |
| File Modified | 0000-00-00 |
| File Created | 2021-01-15 |