SUPPORTING STATEMENT_CBCS Evaluation Ancillary Data Collection_Section A_2-10-17

The Evaluation of the Child Welfare Capacity Building Collaborative, Center for States is sponsored by the Children’s Bureau (CB) in the Administration on Children, Youth and Families (ACYF), Administration for Youth and Families (ACF), U.S. Department of Health and Human Services (DHHS) which seeks approval for the data collection instruments and procedures described herein. The proposed information collection is necessary to facilitate, track, monitor, and evaluate the activities of the Center for States, part of the Capacity Building Collaborative. The Collaborative includes three Federally funded centers (Center for States, Center for Tribes, and Center for Courts) that deliver national child welfare expertise and evidence-informed training and technical assistance services to State, Tribal, and Territorial public child welfare agencies and Court Improvement Programs (CIPs).

The CB funded Centers’ collective goal is to build the capacities of State, local, Tribal child welfare systems to successfully undertake practice, organizational, and systemic reforms necessary to implement federal policies, meet federal standards, and achieve better outcomes for the children, youth and families they serve.

Legislative Background and Purpose

Agencies that receive formula funding through the Child Abuse Prevention and Treatment Act, and titles IV-B and IV-E of the Social Security Act are eligible for technical assistance from the Children’s Bureau to support implementation of these programs, compliance with federal requirements, and improvement of outcomes.

The proposed information collection is necessary to perform routine evaluation of quality and effectiveness and to inform future planning and decision making about the provision and improvement of technical assistance services authorized under multiple sections of the Child Abuse Prevention and Treatment Act (CAPTA) and title IV of the Social Security Act. This information collection also complies with the statutory requirement for projects authorized by Section 5106 of CAPTA to be evaluated for their effectiveness.

A copy of the relevant section of CAPTA can be found in Appendix A.

Purpose and use of the information collection

The Children’s Bureau recently restructured its technical assistance delivery structure and invested in new approaches to service delivery. The Center for States services have been organized into three major categories: (1) product development and information dissemination, including the creation and release of website content, publications, and other resources; (2) training and peer networking, including the delivery of online courses or “learning experiences,” virtual presentations, and facilitated peer discussions; and (3) jurisdiction-specific consultation and coaching, including workshops and onsite visits to states and tribes to provide customized support. Each service category has been designed to achieve specific outcomes that require different levels of engagement and interaction between the Center and its targeted service recipients.

Data collected through this proposed information collection will be used by the Center for States and the Children’s Bureau to improve the development and design of services in each category and evaluation findings with be shared with other providers and service recipients to increase knowledge about technical assistance strategies and approaches. Evaluation findings will also inform future decision making about the service delivery structure and federal resource allocation.

Consistent with this approach, the Children’s Bureau recently released findings from its prior evaluation of 15 training and technical assistance providers to the public. Findings from the final report have been summarized in an executive summary and several briefs to make key information accessible to technical assistance providers, evaluators, and consumers of services. The prior evaluation design has been shared with other federal agencies and departments that fund training and technical assistance systems and results have been presented at national conferences. The Children’s Bureau used data from the prior information collection and findings from its analysis as a basis for the changes to its current service delivery system.

The Center for States’ services will be evaluated by both Center-specific evaluations and a Cross-Center Evaluation. The Center-specific evaluations are designed to collect data on Center-specific processes and outcomes. The Cross-Center Evaluation is designed to respond to a set of cross-cutting evaluation questions posed by the Children’s Bureau. Data collected will address several critical evaluation questions. Research questions for the Cross-Center Evaluation and Center-specific evaluations are provided in Appendix B.

Center for States-specific data sources for this information collection include (1) Instruments to assess the quality and effectiveness of a Child Welfare Virtual Conference; (2) Instruments to assess state experience with Center tailored service delivery in relation to implementation of the Center’s evidence-informed Capacity Building Approach; and (3) questions asked as part of the Center for States Annual Assessment process. This OMB package represents the third in a series of submissions for the Capacity Building Collaborative and these instruments represent Center for States data collection that is necessary to supplement the evaluation activities being conducted by the Collaborative and the Cross-Center evaluation.

Personally Identifiable Information

The personally identifiable information (PII) collected by the Center for States using the Virtual Conference Registration form consist of contact information provided by the registrants. This information will be used to facilitate entry into the conference technology and a tailored participant experience as well as to contact participants who opt to participate in focus groups and interviews following the Virtual Conference to share their experience. Participants who provide contact information will be informed that the information will be used to contact them in the future and that it is completely voluntary. These data are confidential and will only be shared with Center for States staff for facilitation of conference participation and facilitation of focus group participation. All registrants are informed that general demographic data voluntarily captured during registration will be used in aggregate by the Collaborative, Center for States, and the Children’s Bureau. Only aggregate data (e.g., total number of participants with a specific role) will be shared outside of these purposes to determine if the conference reached and engaged the intended audience. To ensure the PII data that is being collected meets required security standards, all PII contact data will be stored in a password-protected secure data system in order to insure privacy. Contact information for focus group participants will be used to secure focus group participation and then destroyed. All other PII contact data will be removed from the database immediately following the virtual conference and destroyed and unique identifiers assigned. A system of records is not being created for this form. No other forms collect personally identifiable information.

Use of improved information technology and burden reduction

Wherever possible and appropriate, information technology will be used to capture information and reduce burden relative to alternative methods of data collection. Administration of any evaluation surveys will be web-based, utilizing email notification and Internet-based survey technologies creating efficiencies for survey administrators, allowing flexibility and convenience for participants, and ideally resulting in a user-friendly experience for respondents. Based on the services provided, survey respondents will receive an email notification inviting them to complete the appropriate survey instrument by accessing a web-link to an online version of the survey.

Nearly all of the targeted respondents are expected to be able to access the web-link to the surveys.¹ The majority of questions in most surveys are closed-ended response items that can be completed quickly (within 5 – 10 minutes), allowing descriptive and comparative analyses.

Child Welfare Virtual Conference Session and Exit surveys are fully web accessible and can be completed in 10 minutes or less. To minimize the burden on Virtual Conference participants, data collection for the Virtual Conference was embedded into the flow of the conference using the Conference technology platform. Reported metrics can be accessed securely through the platform’s data portal. Child Welfare Virtual Conference Focus groups will also be virtual and will utilize the Adobe Connect platform to facilitate effective virtual participation.

Efforts to identify duplication and use of similar information

The proposed instruments are intended to uniformly collect data that will allow for the evaluation of Center for States-specific processes and outcomes. CB has required its cross-center evaluator and center-specific evaluators to ensure data collection is necessary and complementary. The information collection and evaluation activities have been coordinated to avoid potential duplication of data collection and reduce burden to respondents. To reduce burden on respondents, the Tailored Services Practice Model Survey will be embedded into the existing Capacity Survey data collection being conducted by the cross-center evaluators and currently going through OMB approval processes. Additionally, the Assessment and Service Delivery State Lead Interviews – Supplemental Questions will be included in existing Tailored Services Interviews currently going through the OMB approval processes as possible without increasing burden. The questions will then be used as standalone questions as needed to reach the expected sampling burden requested in this approval request. All questions have been cross-referenced to ensure no duplication. By embedding the questions into the existing Capacity Survey data collection or Tailored Services Interviews, no respondent receives more than one survey or interview. Additionally, all instruments in this package have been reviewed for duplication across any of the instruments being reviewed for the Capacity Building Collaborative. The instruments have been revised to address potential overlap and the timing of data collection activities is being closely coordinated to minimize burden.

Impact on small businesses or other small entities

The full range of information will be requested of all respondents. Organizational size will not affect the relevance of particular questions. A number of efforts are in place to minimize respondent burden, regardless of organizational size, for each of the data collection strategies described herein. Skip patterns have been included in the survey instruments based on the types of services received, and the timing of data collection activities is being coordinated to minimize respondent burden. Information being requested has been held to the minimum necessary to respond to the intended evaluation questions.

Consequences of collecting the information less frequently

In order to improve the Center’s services and collaborate effectively to provide coordinated support to State, Tribal, and Territorial public child welfare agencies and Court Improvement Programs, CB and its providers need timely data on the provision of services delivered by the Centers, the accessibility of services, the perceived effect and quality of the services received, and the interactions of service providers with one another. Less frequent data collection would inhibit the timely use of the information by CB and providers to improve service coordination and service quality and to potentially make decisions about resource allocation.

Special circumstances relating to the guidelines of 5 CFR 1320.5

There are no special circumstances associated with this data collection.

Comments in response to the Federal Register Notice and efforts to consult outside the agency

Following publication of the notice that appeared in the Federal Register, Volume 81, Number 229, Tuesday, November 29, 2016, no comments or requests were received from the general public for copies of the proposed information collection instruments.

Explanation of any payment or gift to respondents

No payments or gifts are provided to respondents for completing this information request.

Assurance of confidentiality provided to respondents

No assurances of complete confidentiality are provided to respondents. However, all respondents are informed of the importance of maintaining their privacy and that reported data are aggregated and shared with the Center for States service planning teams and the Children’s Bureau; they are not attributed to individuals.

Justification for sensitive questions

Participants who provide personally identifiable contact information on the Child Welfare Virtual Conference Registration Form are informed that a name and email address is required to facilitate access to the conference or participation in follow up focus groups and will not be shared externally. All registrants are informed that general demographic data voluntarily captured during registration will be used in aggregate by the Collaborative, Center for States, and the Children’s Bureau; therefore there are no opportunities to consent to the sharing of information. To ensure the PII data that is being collected meets required security standards, all PII contact data will be stored in a password-protected secure data system in order to insure privacy. Contact information for focus group participants will be used to secure focus group participation and then destroyed. All other PII contact data will be removed from the database following the virtual conference and destroyed and unique identifiers assigned.

There are no questions or requirements of a sensitive nature contained in the survey instruments and data collection forms described herein.

Estimates of annualized burden hours and costs

Having applied hourly wage estimates to burden hours in each respondent category, the current annual cost to the respondents is as follows: (1) $ 8,367.36 for the Child Welfare Virtual Conference Session Surveys; (2) $653.70 for the Child Welfare Virtual Conference Focus Group Guide; (3) $217.90 for the Child Welfare Virtual Conference Interview Guide. (4) $980.55 for the Child Welfare Virtual Conference Registration Form; (5) $1,743.20 for the Child Welfare Virtual Conference Exit Survey (6) $522.96 for the Tailored Services Practice Model Survey; (7) $621.02 for the Assessment Observation – Group Debrief; (8) $871.60 for the Service Delivery and Tracking Adjustment Observation – Group Debrief; (9) $544.75 for the Assessment and Service Delivery State Lead Interviews – Supplemental Questions; (10) $99.36 for the Annual Assessment Update (only 8 systematic questions).² The total annual cost to the respondents if all data collection instruments were employed in the same given year is $14,622.40.

Exhibit A-1. Annual Burden Estimates

Instrument	Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Total Annual Burden Hours
Child Welfare Virtual Conference
Child Welfare Virtual Conference Session Surveys	800	6	.08	384
Child Welfare Virtual Conference Focus Group Guide	30	1	1	30
Child Welfare Virtual Conference Interview Guide	20	1	.5	10
Child Welfare Virtual Conference Registration Form	1500	1	.03	45
Child Welfare Virtual Conference Exit Survey	500	1	.16	80
Tailored Services Capacity Building Approach
Tailored Services Practice Model Survey	200	1	.12	24
Assessment Observation– Group Debrief	114	1	.25	28.5
Service Delivery and Tracking and Adjustment Observation – Group Debrief	160	1	.25	40
Assessment and Service Delivery State Lead Interviews – Supplemental Questions	50	1	.5	25
Assessment questions
Annual Assessment Update (8 systematic questions)	57	1	.08	4.56
Total				671.06

The Child Welfare Virtual Conference Session Surveys and Child Welfare Virtual Conference Exit Survey will be administered to all participants of the Child Welfare Virtual Conference at the conclusion of each session and the conference. The Child Welfare Virtual Conference Interviews and Child Welfare Virtual Conference Focus Groups will be conducted with a sample of up to 20 (for interviews) and 30 (for focus groups) virtual conference participants after the conference has concluded. The interviews will focus on individual user navigation/experience. The Child Welfare Virtual Conference Registration Form will be completed by all conference registrants.

The Tailored Services Practice Model Survey will be appended to the Cross-Center Evaluation Capacity Building Survey for up to 40 service delivery projects per year. It is anticipated that up to five respondents per project will participate.

The Assessment Observation– Group Debrief will be held at the conclusion of an onsite meeting as part of the assessment process for up to 57 states per year.

A Service Delivery and Tracking and Adjustment Observation – Group Debrief will be held at the conclusion of an onsite meeting as part of the service delivery process up to two times for 40 projects per year.

The Assessment and Service Delivery State Lead Interviews - Supplemental Questions will be conducted with a key stakeholder per state for up to 50 states per year

The Annual Assessment Update will include 8 systematic questions verbally asked of a representative from child welfare agency staff for up to 57 States/territories during the course of the Center for States Annual Assessment process.

Estimates of other total annual cost burden to respondents and record keepers

No additional cost burden will apply for respondents or record keepers.

Annualized cost to the Federal government

The associated costs for administering the surveys are outlined in Table A-2 below. The annual cost to the Federal government for administration is (1) $2,077.92 for the Child Welfare Virtual Conference Session Surveys; (2) $692.64 for the Child Welfare Virtual Conference Focus Group Guide; (3) $4,040.40 for the Child Welfare Virtual Conference Interview Guide, (4) $288.60 for the Virtual Conference Registration Form; (5) $577.20 for the Child Welfare Virtual Conference Exit Survey ; (6) $9,235.20 for the Tailored Services Practice Model Survey; (7) 8225.10 for the Assessment Observation – Group Debrief; (8) $11,544.00 for the Service Delivery and Tracking Adjustment Focus Observation – Group Debrief; (9) $15,873.00 for the Assessment and Service Delivery State Lead Interviews – Supplemental Questions; (10) $592.21 for the Annual Assessment Update (8 systematic questions). The total annual cost to the Federal Government if all data collection instruments were employed in a given year is $53,146.27.

Exhibit A-2. Annualized Costs for Survey Administration

Instrument	Administration Activities	Staff Time	Total Cost
Child Welfare Virtual Conference Session Surveys	Distribute Electronic Survey	36	$2,077.92
Child Welfare Virtual Conference Focus Group Guide	Conduct Focus Groups	12	$692.64
Child Welfare Virtual Conference Interview Guide	Conduct Interviews	70	$4,040.40
Child Welfare Virtual Conference Registration Form	Online registration	5	$288.60
Child Welfare Virtual Conference Exit Survey	Distribute Electronic Survey	10	$577.20
Tailored Services Practice Model Survey	Distribute Electronic Survey & Follow up	160	$9,235.20
Assessment Observation – Group Debrief	Conduct group debrief	142.5	$8,225.10
Service Delivery and Tracking and Adjustment Observation – Group Debrief	Conduct group debrief	200	$11,544.00
Assessment and Service Delivery State Lead Interviews – Supplemental Questions	Conduct Interviews	275	$15,873.00
Annual Assessment Update	Distribute Electronic Survey & Follow up	10.26	$592.21
Total			$ 53,146.27

Explanation for program changes or adjustments

Not applicable.

Plans for tabulation and publication and project time schedule

Tabulation: Frequency distributions will be calculated to generate summaries of survey items, as well as to examine variability in the data. Parameter estimates, such as variances and means, will be established for each quantitative item. Cross-tabulations and significance tests will be conducted as appropriate. Content analysis will be conducted on open-ended survey items and interview transcripts, and will entail systematic coding, creation of a hierarchy of codes, and cross-case and cross-source thematic analysis. Analyses will be conducted to determine subgroup variation.

Publication: For Center for States, findings specific to Center for States service delivery will be reported on an ongoing basis to Center Staff and the Children’s Bureau to inform planning and service delivery. Other opportunities for publication, such as conference presentations, will be explored. Reports of research findings will include descriptive analyses, and the implications of the findings.

Project Timetable: The Child Welfare Virtual Conference Registration form will be used for participants to register for the conference. The Child Welfare Virtual Conference Session Surveys will be administered over the course of the virtual conference as sessions conclude. The Child Welfare Virtual Conference Exit Survey will be administered at the end of the conference. The Child Welfare Virtual Conference Focus Group and Child Welfare Virtual Conference Interviews will be administered following the Child Welfare Virtual Conference to a sample of voluntary participants.

The Tailored Services Practice Model Survey will be administered as service projects end/close out throughout the year and the items will be appended to the cross-center evaluator Capacity Survey under current OMB approval to reduce burden on participants. The Assessment Observation – Group Debrief and the Service Delivery and Tracking and Adjustment Observation – Group Debrief will occur at the conclusion of the identified meetings. The Assessment and Service Delivery State Lead Interviews – Supplemental Questions will occur after a state has engaged in the assessment process or when a service project ends.

The Annual Assessment Update Survey will be administered once per year as part of each State’s Assessment Update.

Reason(s) display of OMB expiration data is inappropriate

The OMB expiration date for the information collection will appear on the instruments.

Exceptions to Certification for Paperwork Reduction Act Submissions

No exception is requested to the certification statement identified in Item 19, “Certification for Paperwork Reduction Act Submissions,” of OMB Form 83-I.

1 A hard copy of the surveys will be provided to those who cannot access the surveys online.

2 The annual respondent burden and annualized cost varies by year and depends upon the data collection strategies employed.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	SUPPORTING STATEMENT SECTION A
Author	hafford
File Modified	0000-00-00
File Created	2021-01-22