Part A-SUPPORTING STATEMENT FOR

The Evaluation of the Child Welfare Capacity Building Collaborative is sponsored by the Children’s Bureau (CB) in the Administration on Children, Youth and Families (ACYF), Administration for Youth and Families (ACF), U.S. Department of Health and Human Services (DHHS) which seeks approval for the data collection instruments and procedures described herein. The proposed information collection is necessary to facilitate, track, monitor, and evaluate the activities of the Capacity Building Collaborative which includes three Federally funded centers (Center for States, Center for Tribes, and Center for Courts) that deliver national child welfare expertise and evidence-informed training and technical assistance services to State, Tribal, and Territorial public child welfare agencies and Court Improvement Programs (CIPs).

The CB funded Centers’ collective goal is to build the capacities of State, local, Tribal child welfare systems to successfully undertake practice, organizational, and systemic reforms necessary to implement federal policies, meet federal standards, and achieve better outcomes for the children, youth and families they serve.

Legislative Background and Purpose

Agencies that receive formula funding through the Child Abuse Prevention and Treatment Act, and titles IV-B and IV-E of the Social Security Act are eligible for technical assistance from the Children’s Bureau to support implementation of these programs, compliance with federal requirements, and improvement of outcomes.

The proposed information collection is necessary to perform routine evaluation of quality and effectiveness and to inform future planning and decision making about the provision and improvement of technical assistance services authorized under multiple sections of the Child Abuse Prevention and Treatment Act (CAPTA) and title IV of the Social Security Act. This information collection also complies with the statutory requirement for projects authorized by Section 5106 of CAPTA to be evaluated for their effectiveness.

A copy of the relevant section of CAPTA can be found in Appendix A.

Purpose and use of the information collection

The Children’s Bureau recently restructured its technical assistance delivery structure and invested in new approaches to service delivery. The Centers’ services have been organized into three major categories: (1) product development and information dissemination, including the creation and release of website content, publications, and other resources; (2) training and peer networking, including the delivery of online courses or “learning experiences,” virtual presentations, and facilitated peer discussions; and (3) jurisdiction-specific consultation and coaching, including workshops and onsite visits to states and tribes to provide customized support. Each service category has been designed to achieve specific outcomes that require different levels of engagement and interaction between the Center and its targeted service recipients.

Data collected through this proposed information collection will be used by the Centers and the Children’s Bureau to improve the development and design of services in each category and evaluation findings with be shared with other providers and service recipients to increase knowledge about technical assistance strategies and approaches. Evaluation findings will also inform future decision making about the service delivery structure and federal resource allocation.

Consistent with this approach, the Children’s Bureau recently released findings from its prior evaluation of 15 training and technical assistance providers to the public. Findings from the final report have been summarized in an executive summary and several briefs to make key information accessible to technical assistance providers, evaluators, and consumers of services. The prior evaluation design has been shared with other federal agencies and departments that fund training and technical assistance systems and results have been presented at national conferences. The Children’s Bureau used data from the prior information collection and findings from its analysis as a basis for the changes to its current service delivery system.

The Centers’ services will be evaluated by both Center-specific evaluations and a Cross-Center Evaluation. The Center-specific evaluations are designed to collect data on Center-specific processes and outcomes. The Cross-Center Evaluation is designed to respond to a set of cross-cutting evaluation questions posed by the Children’s Bureau. Data collected will address several critical evaluation questions. Research questions for the Cross-Center Evaluation and Center-specific evaluations are provided in Appendix B.

Proposed Cross-Center Evaluation data sources for this effort include (1) satisfaction surveys to assess recipients’ satisfaction with services, such as the Learning Experiences Satisfaction Survey; (2) a leadership interview, administered to all State child welfare directors, Tribal child welfare directors, and CIP coordinators that are receiving services from the Centers; and (3) a collaboration survey, an annual web-based survey administered to the directors and staff of the three Centers. Center-specific data sources for this effort include (1) assessment tools such as the Tribal Organizational Assessment Caseworker Interview; and (2) service-specific feedback forms, such as the Center for States Intensive Projects instrument and the Center for Courts CQI Workshops instrument. This OMB package represents the first in a series of submissions for this project. These instruments represent the data collection that is necessary in the first phase of the project.

Personally Identifiable Information

Personally identifiable information (PII) from individuals representing Tribes that are seeking technical assistance from the Center for Tribes will be captured using two tools: the Center for States Contact Form and the Needs and Fit Exploration Tool, Phase I. PII will include the following data elements: Name of tribal contact, job title, address, phone number and email address. Contact information will be populated into the Demographic Survey Tool. These data will be gathered for each tribe that makes a request for services so that the service providers, evaluators, and Children’s Bureau have reliable contact information from the requester in order to facilitate further correspondence and to schedule future meetings and service delivery. All PII contact data will be stored in a password-protected secure web-based data system in order to insure privacy.

Use of improved information technology and burden reduction

Wherever possible and appropriate, information technology will be used to capture information and reduce burden relative to alternative methods of data collection. Administration of the majority of evaluation surveys will be web-based, utilizing email notification and Internet-based survey technologies creating efficiencies for survey administrators, allowing flexibility and convenience for participants, and ideally resulting in a user-friendly experience for respondents. Based on the services provided, survey respondents will receive an email notification inviting them to complete the appropriate survey instrument by accessing a web-link to an online version of the survey.

Nearly all of the targeted respondents are expected to be able to access the web-link to the surveys.^¹ The majority of questions in most surveys are closed-ended response items that can be completed quickly (within 10 – 15 minutes), allowing descriptive and comparative analyses. One in-person survey – the CQI Workshop survey, is administered by paper at the conclusion of the workshop. The survey takes fewer than 10 minutes to complete.

Efforts to identify duplication and use of similar information

The proposed instruments are intended to uniformly collect data that will allow for the evaluation of center-specific processes and outcomes and to answer a set of cross-cutting evaluation questions posed by the Children’s Bureau. CB has required its cross-center and center-specific evaluators to ensure data collection is necessary and complementary. The information collection and the center-specific evaluation activities have been coordinated to avoid potential duplication of data collection and reduce burden to respondents. Each of the three Centers have met with the cross-center evaluator reviewed each of the cross-center data collection instruments. The instruments have been revised to address potential overlap and the timing of data collection activities is being closely coordinated to minimize burden. While center-specific data will yield important and relevant information it will not be sufficient to meet the cross-center purposes for the proposed information collection.

Impact on small businesses or other small entities

The full range of information will be requested of all respondents. Organizational size will not affect the relevance of particular questions. A number of efforts are in place to minimize respondent burden, regardless of organizational size, for each of the data collection strategies described herein. Skip patterns have been included in the survey instruments based on the types of services received, and the timing of data collection activities is being coordinated to minimize respondent burden. Information being requested has been held to the minimum necessary to respond to the intended evaluation questions.

Consequences of collecting the information less frequently

In order to improve the Center’s services and collaborate effectively to provide coordinated support to State, Tribal, and Territorial public child welfare agencies and Court Improvement Programs, CB and its providers need timely data on the provision of services delivered by the Centers, the accessibility of services, the perceived effect and quality of the services received, and the interactions of service providers with one another. Less frequent data collection would inhibit the timely use of the information by CB and providers to improve service coordination and service quality and to potentially make decisions about resource allocation.

Special circumstances relating to the guidelines of 5 CFR 1320.5

There are no special circumstances associated with this data collection.

Comments in response to the Federal Register Notice and efforts to consult outside the agency

Following publication of the notice that appeared in the Federal Register, Volume 80, Number 211, Monday, November 2, 2015, page 67407-67408, no requests were received from the general public for copies of the proposed information collection instruments.

However, numerous opportunities were provided for direct stakeholders to review the proposed instruments and to contribute to their development throughout the design phase of this study. The Cross-Center and Center evaluation teams have made an effort to be responsive to stakeholders’ comments whenever possible and have used their feedback in revising the data collection instruments. Instruments were pilot-tested with individuals who were knowledgeable of the topics addressed and who had served in positions similar to the potential respondents (i.e, State/Tribal Child Welfare Directors, CIP Directors, State technical assistance liaisons, current and former Center staff members and consultants). Following stakeholder review and pilot testing, revisions were made to instruments based on comments to improve clarity of instructions and items and, in some cases, to shorten the instruments. The Collaboration Survey was revised to further clarify instructions for the subsection of the Collaboration Survey for which there are plans to do a network analysis; and in response to a stakeholder comment that the response scale did not capture both the intensity and frequency of collaboration. None of these revisions have added to the burden of completing the instruments and forms.

Explanation of any payment or gift to respondents

No payments or gifts are provided to respondents for completing this information request.

Assurance of confidentiality provided to respondents

No assurances of complete confidentiality are provided to respondents. However, all respondents are informed of the importance of maintaining their privacy and that reported data are aggregated; they are not attributed to individuals.

Justification for sensitive questions

Tribes that provide personally identifiable contact information are informed that participation in services and the provision of contact information are completely voluntary. Contacts will be informed that their information will only be used to facilitate correspondence related to service delivery. Since this contact data is not shared externally outside of the Capacity Building Collaborative, evaluation contract, and Children’s Bureau, there are no opportunities to consent to the sharing of information. All PII contact data will be stored in a password-protected secure web-based data system in order to insure privacy. A system of records is not being created for these forms.

There are no questions or requirements of a sensitive nature contained in the survey instruments and data collection forms described herein.

Estimates of annualized burden hours and costs

Having applied hourly wage estimates to burden hours in each respondent category, the current annual cost to the respondents is as follows: (1) $18,151.07 for the Satisfaction Surveys; (2) $4,989.91 for the Center for Tribes Forms, Demographic Survey, and Tribal Assessment Interviews; (3) $11,439.75 for the Center for States Forms; (4) $740.86 for the CIP Survey and Center for Courts CQI Workshops; (5) $7,360.44 for the Leadership Interviews; and (6) $4,684.52 for the Collaboration Survey.^² The total annual cost to the respondents if all data collection instruments were employed in the same given year is $48,140.10.

Exhibit A-1. Annual Burden Estimates

Instrument	Annual Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Total Annual Burden Hours
Webpages and Products Satisfaction Survey	1,560	1	.08	125
Learning Experiences Satisfaction Survey (single)^³	500	1	.33	165
Learning Experiences Satisfaction Survey (intensive)^⁴	900	1	.08	72
Webinars, Events, and In-Person Meetings Satisfaction Survey	5,502	1	.08	441
Assessment & Capacity Building Plan Satisfaction Survey	450	1	.066	30
Center for Tribes Contact Form	50	1	.05	3
Center for Tribes Demographic Survey	20	1	1.75	35
Center for Tribes Needs and Fit Exploration Tool Phase 1	30	1	1.5	45
Center for Tribes Needs and Fit Exploration Tool Phase 2	25	1	3.0	75
Tribal Organizational Assessment Caseworker Interview	20	1	1.25	25
Tribal Organizational Assessment Community Provider Interview	16	1	1.25	20
Tribal Organizational Assessment Community Member/Elder Interview	12	1	1.0	12
Tribal Organizational Assessment Family Interview	14	1	1.0	14
Center for States Information and Referral Survey	12	1	.05	1
Center for States Intensive Projects Survey	330	2	.33	218
Center for States Constituency Groups Surveys	400	2	.33	264
Center for States Brief Tailored Services Survey	125	1	.33	42
CIP Annual Meeting Survey	200	1	.13	26
Center for Courts CQI Workshops	48	1	.17	8
Leadership Interview – States and Territories	13	2	1	26
Leadership Interview – CIPs	13	2	1	26
Leadership Interview – Tribes	8	2	1.25	20
Leadership Interview Part II – Tribes	8	2	.67	11
Annual Collaboration Survey	230	1	.36	83
Total				1,787

Estimates of other total annual cost burden to respondents and record keepers

No additional cost burden will apply for respondents or record keepers.

Annualized cost to the Federal government

The associated costs for administering the surveys are outlined in Table A-2 below. The annual cost to the Federal government for administration is (1) $50,562.72 for the Satisfaction Surveys; (2) $42,539.64 for the Center for Tribes Forms; (3) $27,082.22 for the Center for States Forms; (4) $8,080.80 for the CIP Survey and Center for Courts CQI Workshops; (5) $9,696.96 for the Leadership Interviews; and (6) $2,886.00 for the Collaboration Survey.^⁵ The total annual cost to the respondents if all data collection instruments were employed in the same given year is $140,848.34.

Exhibit A-2. Annualized Costs for Survey Administration

Instrument	Administration Activities	Staff Time	Total Cost
Webpages and Products Satisfaction Survey	Distribute electronic survey & follow up	264	$15,238.08
Learning Experiences Satisfaction Survey (single)^⁶	Distribute electronic survey & follow up	52	$3,001.44
Learning Experiences Satisfaction Survey (intensive)^⁷	Distribute electronic survey & follow up	72	$4,155.84
Webinars, Events, and In-Person Meetings Satisfaction Survey	Distribute electronic survey & follow up	308	$17,777.76
Assessment & Capacity Building Plan Satisfaction Survey	Distribute electronic survey & follow up	180	$10,389.60
Center for Tribes Contact Form	Distribute electronic survey & follow up	150	$8,658.00
Center for Tribes Demographic Survey	Distribute electronic survey & follow up	100	$5,772.00
Center for Tribes Needs and Fit Exploration Tool Phase 1	Distribute electronic survey & follow up	120	$6,926.40
Center for Tribes Needs and Fit Phase 2	Conduct interview	150	$8,658.00
Tribal Organizational Assessment Caseworker Interview	Conduct interview	70	$4,040.40
Tribal Organizational Assessment Community Provider Interview	Conduct interview	56	$3,232.32
Tribal Organizational Assessment Community Member/Elder Interview	Conduct interview	42	$2,424.24
Tribal Organizational Assessment Family Interview	Conduct interview	49	$2,828.28
Center for States Information and Referral Survey	Distribute electronic survey & follow up	4.2	$242.42
Center for States Intensive Projects Survey	Distribute electronic survey & follow up	165	$9,523.80
Center for States Constituency Group Survey	Distribute electronic survey & follow up	200	$11,544.00
Center for States Brief Tailored Services Survey	Distribute electronic survey & follow up	100	$5,772.00
CIP Annual Meeting Survey	Distribute electronic survey & follow up	50	$2,886.00
Center for Courts CQI Workshops	Distribute electronic survey & follow up	90	$5,194.80
Leadership Interviews – (States, Territories, CIPs, and Tribes)	Conduct interview	168	$9,696.96
Annual Collaboration Survey	Distribute electronic survey & follow up	50	$2,886.00
Total			$140,848.34

Explanation for program changes or adjustments

Not applicable.

Plans for tabulation and publication and project time schedule

Tabulation: Frequency distributions will be calculated to generate summaries of survey items, as well as to examine variability in the data. Parameter estimates, such as variances and means, will be established for each quantitative item. Cross-tabulations and significance tests will be conducted as appropriate. Content analysis will be conducted on open-ended survey items and interview transcripts, and will entail systematic coding, creation of a hierarchy of codes, and cross-case and cross-source thematic analysis. Analyses will be conducted to determine subgroup variation.

Publication: The findings from the annual information collections will be summarized and tabulated in a series of annual briefings and reports to the CB beginning in the first year of data collection in FY 2015. For Center for States, findings specific to Center for States service delivery will be reported on an ongoing basis to Center Staff to inform planning and service delivery. Reports of research findings will include descriptive analyses, and the implications of the findings. In the last stage of analysis beginning in 2018, data will be merged from multiple sources to enable final summative analyses to address major questions on the cumulative, overall results of the three centers. A final synthesis report of the project’s findings for all years will be submitted to the CB in FY 2018 for dissemination to federal, state and tribal stakeholders.

Project Timetable: The Webpages and Products, Learning Experiences, and Webinars, Events, and In-Person Meetings surveys will be administered by Centers on an ongoing basis as services are delivered to individuals and groups of child welfare professionals throughout the project period. The Assessment and Capacity Building Plan Satisfaction Survey will be administered annually upon the Center’s completion of the assessment and work planning process with public child welfare agencies.

The Center for Tribes Contact Form allows tribes to complete and send a short form describing a request for services. A tribe can access an electronic version through a website and submit it to the Center or, meeting/conference attendees may pick up a paper version, complete, and return the form in-person to a Center representative. The Needs and Fit Exploration Tool (NAFET) Phase I is used to gather additional information about the request through a structured telephone interview with the tribal representative who made the request. If the initial request is approved, the Center for Tribes conducts an onsite visit to conduct a structured interview with tribal representatives using the Needs and Fit Exploration Tool (NAFET) Phase 2 form. The Demographics Survey tools is also administered during this visit to gather information about the structure and operations of the Tribal Child Welfare Program. The Tribal Organizational Assessment Caseworker Interview, Tribal Organizational Assessment Community Provider Interview, Tribal Organizational Assessment Community Member/Elder Interview, Tribal Organizational Assessment Family Interview will be administered when a tribe requests and becomes engaged with the Center in “tailored” services that frequently involve site visits and individualized consultation from Center staff over a period of months.

The Center for States Information and Referral Survey will be administered to child welfare professionals after each information request that results in the provision of information to the requestor and referral by the Center to additional resources or services. The Center for States Intensive Projects Survey will be administered twice for every intensive “tailored” services engagement between the Center and a state that lasts for more than 12 months. The Intensive Projects Survey will be administered once three months after services begin and once after 12 months. If the intensive project continues for subsequent years, this survey will be administered again using the same time frame. The Center for States Constituency Group Survey will be administered to every “constituency group” (a peer group of child welfare professionals – e.g., adoption manager, foster care managers) served by the Center, every six months after constituency group activities and services are launched. The Center for States Brief Tailored Services Survey will be administered twice for every brief service (tailored consultation between the Center and an individual state lasting fewer than 12 months), once after services begin and once after services are closed.

The CIP Annual Meeting Survey will be administered annually following the Annual CIP Meeting. The Center for Courts CQI Workshops survey will be administered approximately three times per year following all CQI Workshops.

The Leadership Interviews will be administered in years FY 2016 and 2018. The Collaboration Survey will be administered once per year in FY 2016, 2017, and 2018.

Reason(s) display of OMB expiration data is inappropriate

The OMB expiration date for the information collection will appear on the instruments.

Exceptions to Certification for Paperwork Reduction Act Submissions

No exception is requested to the certification statement identified in Item 19, “Certification for Paperwork Reduction Act Submissions,” of OMB Form 83-I.

1 A hard copy of the surveys will be provided to those who cannot access the surveys online.

2 The annual respondent burden and annualized cost varies by year and depends upon the data collection strategies employed.

3 For Learning Experiences that consist of a single event (e.g. on-line session or in-person training)

4 For more intensive Learning Experiences that require administration of multiple surveys over a series of events, modules, or units

5 The annual respondent burden and annualized cost varies by year and depends upon the data collection strategies employed.

6 For Learning Experiences that consist of a single event (e.g. on-line session or in-person training)

7 For more intensive Learning Experiences that require administration of multiple surveys over a series of events, modules, or units

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Windows User
File Modified	0000-00-00
File Created	2021-01-24