Evaluation of the Child Welfare Capacity Building Collaborative: Part Two
Supporting Statement Part B
Department of Health and Human Services
Children’s Bureau
Washington, DC
Contact person:
Brian Deakins
Children’s Bureau
Administration on Children, Youth and Families
Mary E. Switzer Building
3rd Floor, Mailstop 3602
330 C Street, SW
Washington, DC 20201
202-205-8769
Section B: Collections of Information Employing Statistical Methods
Overview
The Evaluation of the Child Welfare Capacity Building Collaborative: Part Two is sponsored by the Children’s Bureau (CB) in the Administration on Children, Youth and Families (ACYF), Administration for Youth and Families (ACF), U.S. Department of Health and Human Services (DHHS) which seeks approval for the data collection instruments and procedures described herein. The proposed information collection is necessary to facilitate, track, monitor, and evaluate the activities of the Capacity Building Collaborative which includes three federally funded centers (Center for States, Center for Tribes, and Center for Courts) that deliver national child welfare expertise and evidence-informed training and technical assistance services to State, Tribal, and Territorial public child welfare agencies and Court Improvement Programs (CIPs).
The CB-funded Centers’ collective goal is to build the capacities of State, local, Tribal child welfare systems and CIPs to successfully undertake practice, organizational, and systemic reforms necessary to implement federal policies, meet federal standards, and achieve better outcomes for the children, youth and families they serve.
There are two parts to the evaluation of the Child Welfare Capacity Building Collaborative.
Part One: Part One includes information collection approved under OMB #0970-0484. A request for an extension with minor revisions was submitted for review on August 19, 2019.
Part Two: This information collection request (ICR) is specific to the second part of the data collection for the evaluation. This ICR is for three additional years of approval and includes the following:
Removal of three approved instruments: Capacity Survey, P.L. 113-183 Key Informant Survey, and P.L. 113-183 Key Informant Interview. No additional data collection is needed for the PL 113-183 instruments as the timeline for implementation of the law has passed. We plan to submit a revised Capacity Survey for approval at a later date following input from stakeholders of the Child Welfare Capacity Building Collaborative.
Addition of one new instrument (Center for States Innovation Survey).
Extension of approval for all other previously approved instruments.
The universe of respondents includes participants in or recipients of capacity building services provided by the Collaborative. Each Center serves a specific target audience. The Center for States supports state and territorial public child welfare agencies; Center for Tribes serves title IV-B and title IV-E tribal child welfare agencies and organizations, and Center for Courts provides services to state and tribal Court Improvement Programs (CIPs).
No statistical methodology for stratification and sample selection will be used for any of the data collection instruments aside from the Center for States Tailored Services Interview. For this instrument, a purposive sample of up to 12 States will be selected based upon geographic and services content specific variables such as regional location, length of service, topic of service in relation to Center priorities, and type of outcomes achieved.
The procedures for the collection of information for each instrument is described below.
Foundational Assessment Survey: This survey will be administered at most annually to all state, territory, and tribal child welfare staff identified by the Center for States and Center for Tribes as a participant in the assessment process. Administration frequency will depend upon a state/tribe’s decision to engage in assessment with the Center and the timing of the assessment. The Foundational Assessment Survey will be administered to all CIPs following the National CIP Meeting, which is held annually.
Continuous Quality Improvement (CQI) Workshop Follow-Up Survey: This survey will be administered to members of CIP teams that attend a CQI Workshop. The survey will be administered 6 months after the Workshop and again at 12 months after the Workshop.
Tailored Services Satisfaction Survey: The Tailored Services Satisfaction Surveys will be administered to State, territory, and Tribal child welfare agency staff and CIP staff who participate in tailored services with Centers. For Center for States and Center for Tribes, the surveys will be administered to jurisdiction staff who engage in intensive Center services. The Tailored Services Satisfaction Survey will be administered to CIPs one year following their participation in CQI Workshops.
CapLEARN Registration: This form will be completed once by all individuals who create an account to access the Collaborative resource library and online trainings.
Webinar Registration: This collection will occur for all State and territory public child welfare agency staff, CIP staff, federal staff, and other stakeholders who register to participate in Center webinars.
Center for Courts Universal and Constituency Services Survey: The target population for the Center for Courts Universal and Constituency Services survey is all CIP Directors and Coordinators. The survey will be administered annually via a web link.
Center for States Tailored Services Interview: These interviews, or focus groups, will be conducted with a sample of up to 12 State and territory public child welfare agency staff upon completion of brief or intensive projects.
Center for States Assessment and Work Planning Survey: This survey will be administered every other year to half of the States, alternating with the Cross-Center Satisfaction with Assessment and Work Planning Survey.
Center for States Innovation Survey: This survey will be administered twice per year to a total of 50 jurisdiction staff (25 staff per administration) who participate in new service strategies or innovative techniques.
Maximizing response rates is critical to the administration of the aforementioned surveys. The content and format of the instruments were developed in close consultation with key stakeholders, including the Collaborative Centers and were informed by previously developed measures involving technical assistance provision.
Strategies that emphasize flexibility, privacy, and a respect for the respondent’s time facilitate timely participation. The following strategies1 will be implemented to maximize participation in the data collection and achieve the desired response rates of above 35 percent, which published organizational studies have shown are typical for surveys that use key informants and online data collection.2
Introduction and notification: Strategies to introduce and notify respondents about data collection are used for several instruments. For the Center for Courts, all recipients of brief and intensive services, learning experiences, peer networking events, and constituency groups, will be notified at the beginning of service delivery of the Center’s evaluation efforts and the desire for their participation. For Center for States Tailored Services Interviews and the Innovation survey, potential participants will be informed of the administration and participation requested at least one month prior to administration. They will have the opportunity to ask any questions and will be informed their feedback is critical to planning for future service provision and understanding service effectiveness. For the Center for States Assessment and Workplanning Survey, participants will be informed during initial messaging regarding the assessment and workplanning process about the survey and how the information will be used to inform service planning and development along with understanding service effectiveness. An additional description about the survey and participation will be included in the message distributed with the survey.
For the Webinar Registration Form, participants are notified of the opportunity to register for each webinar with the initial invitation and through webinar marketing efforts. Upon entry into the CapLEARN system, details regarding data to be collected are provided in the form. Participants encounter the Collaborative’s CapLEARN registration form on the Collaborative’s website as they set up their account to track their progress.
The Foundational Assessment Survey will be introduced to States, Tribes, and CIPs first through discussion and a memorandum of understanding that indicates participation in the Cross-Center Evaluation by each jurisdiction receiving training and technical assistance from a collaborative Center is anticipated. Each Center will introduce the Foundational Assessment Survey as part of its assessment process with the jurisdiction. An introductory email with a survey link then will be sent from the Cross-Center Evaluation team to assessment participants from each jurisdiction identified by the Center. Reminder emails will be sent to all State, CIP, and Tribal recipients. We recognize that there may be some turnover in the assessment participants over time and that Centers may have to re-introduce the survey. The CQI Workshop Follow-Up Survey will be introduced and described by Center for Courts’ staff to attendees of the CQI Workshops.
Appendix C includes copies of the introductory emails and reminders to be used with the data collection instruments included in this information collection request.
Timing of data collection: Discussions were held with stakeholders to determine optimal periods for data collection in order to minimize respondent burden and to facilitate recall.
The Foundational Assessment Survey will be completed in alignment with the assessment process, approximately one month prior to the start of the annual assessment for each jurisdiction receiving training and technical assistance. For CIPs, the Foundational Assessment Survey will be administered annually following the National CIP Meeting. The Tailored Satisfaction Services Survey will be administered at the end of intensive tailored services or one year after attendance at the CQI Workshops. The CQI Workshop Follow-Up Survey will be administered 6 months after respondents attend a CQI Workshop and again at 12 months after the Workshop in conjunction with the Tailored Services Satisfaction Survey.
The CapLEARN Registration and Webinar Registration forms will be administered throughout the project period following the administration guidelines described in section B-1. CapLEARN Registration is available online at all times for an individual to make an account to access the resource and training library.
The Center for Courts Universal and Constituency Services Survey will be completed annually at a scheduled time to follow within six months of CIP annual reporting efforts.
The Center for States interviews will be conducted annually, if needed, and the Assessment and Workplanning Survey will be administered at the end of assessment and workplanning service delivery in in each year. The Center for States Innovation Survey will be administered twice per year following the receipt of new service strategies or innovative techniques.
Pre-interview preparation: A copy of the Center for States Tailored Interviews will be sent to respondents in advance of the telephone interview.
Administration: An email invitation will be sent to the appropriate participants of the Foundational Assessment Survey, Tailored Satisfaction Services Survey, and CQI Workshop Follow-Up Survey with a request to complete the survey (i.e., by accessing a web-link to an online version of the survey or accessing an attached survey to complete and return via email, mail or secure fax). Weekly reminder emails will be sent for two weeks to respondents who have not yet completed the surveys. For the Webinar Registration, announcements will be made via listservs, social media, and email communication about upcoming webinars and include registration links. The CapLEARN registration is open at all times to any individual to create an account. For the Center for Courts Universal and Constituency Services survey, all CIP Coordinators and Directors will receive an email from the Center for Courts evaluation team asking them to please participate in the survey. They will be provided an email link to take the survey electronically. A follow-up email reminder will be sent out after one week to increase response rates. The Center for States Tailored Services Interview will be scheduled through tailored services liaisons when needed. The Assessment and Workplanning survey and the Innovation Survey will be distributed via email to complete the survey. Reminder messages will be provided, when necessary.
Alternate response methods: Respondents will be given the option to use an alternate method for the Foundational Assessment Survey and Tailored Services Interviews. For the Tailored Services Interviews if a respondent prefers to submit written responses to a survey in lieu of participating in a telephone interview, the Center for States will provide a paper version to complete and submit via fax, email, or mail. Similarly, paper versions of any of the electronic surveys will be sent to respondents upon request or will be administered through a telephone interview if requested to accommodate any special needs.
Assurances of data privacy: Respondents to all surveys will be assured that reported data are aggregated and not attributable to individuals or organizational entities.
The Cross-Center evaluation instruments contained herein were subject to review and feedback by key stakeholders, including the CB, and staff from all three Centers. The Center for States and the Center for Courts’ instruments were reviewed by key stakeholders, the CB, and Center staff.
Each of the instruments in this information collection request was pilot tested with 9 or fewer individuals (i.e., former CIP directors, former child welfare agency personnel) to confirm survey item validity and to identify possible procedural or methodological challenges in need of attention or improvement. Following the pilot tests, the instruments were refined to minimize burden and improve utility. The pilot tests were instrumental in determining the amount of time required to complete the surveys and forms and develop the burden estimates. Our experiences with the Cross-Center instruments to date indicate respondents have been able to answer the questions and complete all sections. Respondents have not provided any comments or feedback regarding any items and have been able to follow the directions as anticipated.
National Cross-Center Evaluation Contractor |
|
James Bell Associates 1001 19th Street North, Suite 1500 Arlington, VA (703) 528-3230 |
ICF International 9300 Lee Highway Fairfax, VA 22030 |
Capacity Building Center for States |
|
ICF International 9300 Lee Highway Fairfax, VA 22030 |
|
Capacity Building Center for Courts |
|
National Council of Juvenile and Family Court Judges PO Box 8970 Reno, NV 89507 (775)-784-4883 |
|
1 Strategies that pertain to two or more data collections are discussed together.
2 Fulton, B. (2018). Organizations and survey research. Sociological Methods & Research, 47(2), 240-276.
Baruch, Y. & Holtom, B.C. (2008). Survey response rate levels and trends in organizational research. Human Relations, 61(8), 1139-1160.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Jones, Molly (ACF) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-15 |