SCWS-Supporting Statement-A_fnl 9.8.23

Strengthening Child Welfare Systems to Achieve Expected Child and Family Outcomes

Evaluation

OMB Information Collection Request

0970 – New Collection

Supporting Statement Part A - Justification

September 2023

Submitted By:

Children’s Bureau

Administration for Children and Families

U.S. Department of Health and Human Services

1. Circumstances Making the Collection of Information Necessary

In 2018, the Children’s Bureau (CB) awarded five demonstration projects through the Strengthening Child Welfare Systems (SCWS) to Achieve Expected Child and Family Outcomes grant cluster for the development, implementation, and evaluation of strategies that improve social work and other organizational practices and improve permanency and other child welfare outcomes as measured by the Child and Family Service Reviews (CFSRs). Through this grant opportunity, CB contracted with James Bell Associates (JBA) to conduct a study to understand the experiences of the SCWS grant recipients in the implementation of their grant interventions and learn how implementation activities supported or hindered the grant recipients’ ability to achieve expected outcomes related to child safety, permanency, and well-being. While the federal contract with JBA utilizes the term “cross-site evaluation,” this study is limited in its scope and is not using common measures across the grant cluster to determine impact, as would be the case with a traditional cross-site evaluation and therefore we do not use this term throughout¹.

This information collection is related to and funded by CB, and is authorized by Statutory Authority Title II, Section 203(b)(4) of the Child Abuse Prevention and Treatment and Adoption Reform Act of 1978 (42 U.S.C. 5113(b)(4)).

2. Purpose and Use of the Information Collection

The study team will collect information from SCWS grant recipients and conduct a comprehensive document review to (1) understand the implementation processes of grant interventions and (2) examine whether and the degree to which grant recipients were able to address common CFSR safety, permanency, and well-being outcomes. Through the data collected for the study, CB expects to have a better understanding of the factors associated with the successful implementation of grant interventions, whether interventions were successful in improving targeted outcomes, and the ability of grant recipients to sustain the interventions.

Data collection includes the following:

1. The SCWS Grant and Partner Survey will gather information from grant recipient staff (project directors/managers and staff), evaluators, and community agency partners (agency directors/managers and staff) to understand the factors that supported or hindered implementation and assess collaboration and its effect on grant interventions. This survey consists of 10 open-ended questions and 36 multiple choice questions that cover five core areas: respondent characteristics, level of collaboration, implementation, impact (e.g., ability to improve outcomes), and other factors (e.g., opportunities provided by grant, lessons learned). The survey includes skip patterns based on respondent type.

2. The SCWS Grant Focus Groups will gather information from grant recipient staff (project directors/managers and staff), evaluators, and community agency partners (agency directors/managers and staff) to understand implementation of grant strategies and interventions, successes and challenges, and the perceived effect of the strategies on short- and long-term child welfare outcomes, with specific attention to CFSR outcomes related to permanency. The focus group protocol consists of 13 questions divided into three core areas: process, impact (e.g., meeting goals of grant, ability to improve outcomes), and other (e.g., sustainability of interventions, lessons learned).

Additionally, the study team will analyze several grant recipient deliverables and conduct a comprehensive document review. Grant recipients are required to submit semi-annual progress reports to CB providing information for each six-month period on the activities and accomplishments for each intervention implemented, problems encountered, evaluation activities and findings, dissemination activities, and activities planned for the next reporting period². For their project, each grant recipient is conducting an evaluation and is required to submit a final evaluation report at the conclusion of the grant period. The final reports will provide information on the problem(s) being addressed and goals of the project; interventions implemented including target population and outcomes of interest; evaluation design, research questions, process and outcome findings, challenges, and limitations; sustainability of project interventions; and practical recommendations for CB.

The information collected will help explain how the implementation of grant interventions, including efforts to improve child welfare practice, contributes to changes in CFSR and other child and family outcomes. It is not intended to be used as the principal basis for any decision about grants by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.

Information gathered will be summarized in a report for CB and disseminated to grant recipients and the public through CB and JBA websites. The report will provide information on grant interventions, implementation successes and challenges, factors that influenced implementation (e.g., collaboration with partners, the COVID-19 pandemic), and achievement of outcomes of interest, including CFSR outcomes, and lessons learned.

3. Use of Improved Information Technology and Burden Reduction

The SCWS Grant and Partner Survey will be administered using Qualtrics, a web-based survey software. Respondents will be sent a link to access the survey and will have the option to complete it using their computers, tablets, or smart phones. The study team will test the survey link from each type of device to make sure that the survey displays properly and that responses are accurately recorded by the Qualtrics software. Utilizing web-based software will create efficiencies for survey administration and allow for flexibility and convenience for respondents. If for any reason a respondent is unable to utilize an electronic platform for responding, a hard copy of the survey can be provided. 

The SCWS Grant Focus Groups will be conducted virtually through the Zoom web-based platform, which will reduce the burden for travel to a central location and reduce the overall time needed for participation. The focus groups will also be scheduled at a time that is convenient for the participants. Utilizing the Zoom platform provides the option to conduct the focus groups by telephone if an internet connection is not available to the participants.

4. Efforts to Identify Duplication and Use of Similar Information

As a requirement of the grant, each recipient submits to CB semi-annual progress reports (OMB #: 0970-0406; ACF Performance Progress Report – Program Indicators) and will also submit a final report based on findings from their grant-specific evaluations. These data sources will be used to supplement the proposed data collection effort. While these data sources provide rich information, the proposed data collection is necessary to sufficiently understand the perspectives of the grant recipients about the implementation processes and the successes and challenges faced in attempting to achieve targeted CFSR outcomes. However, review of the available information from grant recipients has allowed us to limit the number of questions in the focus group protocol and survey instrument and limit the need for additional data collection activities, therefore reducing burden on respondents.

5. Impact on Small Businesses or Other Small Entities

The proposed information collection will not have a significant impact on small businesses or other small entities. The grant recipients include two universities, one state child welfare agency, one state judiciary, and one non-profit child welfare agency. However, the grant community partners who will be asked to participate in data collection may include small businesses. The total proposed data collection of 60 survey participants and 30 focus group participants is across all five grant recipients—a total of 12 survey respondents and 6 focus group participants from each grant recipient, including grant project staff, the grant evaluator, and community partner agency staff.

6. Consequences of Collecting the Information Less Frequently

To reduce response burden, the proposed approach to data collection limits the scope of the survey and focus groups to the information needed for the study and limits data collection to one response per respondent. If this information were not collected, the study would lack the information necessary to understand the key activities and components required for the successful implementation of the grant recipients’ interventions and their ability to effect change on key outcomes of interest to CB.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

There are no special circumstances relating to 5 CFR 1320.5.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on June 1, 2023, Volume 88, Number 105, page 35884, and provided a sixty-day period for public comment. During the notice and comment period, no comments were received.

No outside consultation was requested for development of the study instruments.

9. Explanation of Any Payment or Gift to Respondents

No incentives for respondents are proposed for this information collection.

10. Assurance of Confidentiality Provided to Respondents

Respondents will be informed of all planned uses of the data, that their participation is voluntary, and that their information will be kept private. No assurance of confidentiality will be provided to respondents. The study team will not disclose any individual-level survey or focus group information to the persons outside the study team. Information will not be published that could be used to identify individual respondents or participating agencies. All survey results will be analyzed and reported in the aggregate for the final report.

All study materials, including all instruments, protocols, and consent documents, will be reviewed, and approved by an Institutional Review Board (IRB) prior to initiation of the study.

The study team will use an encrypted folder designated for this project in a Microsoft Office 365 SharePoint Site. Additionally, JBA maintains a set of security policies and procedures that ensure that JBA’s user responsibility is maintained. Focus groups will be conducted through Zoom and survey data collection will occur through Qualtrics. JBA has an enterprise license for the version of Qualtrics which has FedRAMP certification.

11. Justification for Sensitive Questions

The proposed information collection does not include sensitive information.

12. Estimates of Annualized Burden Hours and Costs

Estimated Burden Hours for Respondents

There are currently five SCWS grant recipients. We anticipate a total of 12 survey respondents and 6 focus group participants from each grant recipient, for a total of 60 survey respondents and 30 focus group participants. Participants will include grant project staff, the grant evaluator, and community partner agency staff.

The estimated times per response for respondents were obtained through pilot testing of the instruments by the study team. Two JBA staff members with knowledge of the grant recipients and one staff person from another organization (ICF, Inc.) who works with the SCWS grant recipients on implementation activities but who is otherwise not involved in this study, pilot tested the survey and participated in a mock focus group. This pilot process was intended to ensure that the content of the survey instrument and focus group protocol is clear and to estimate the time (in minutes) required to complete each of the data collection activities. The average time estimates shown below represent the mean number of minutes required for each data collection activity across the pilot testers.

Table 1: Estimated Annualized Burden Hours and Cost

Information Collection Title	Total Number of Respondents	Total Number of Responses Per Respondent	Average Burden Hours Per Response	Total/Annual Burden Hours	Average Hourly Wage	Total Annual Cost
SCWS Grant and Partner Survey	60	1	.5	30	$73.90	$2,217.00
SCWS Grant Focus Groups	30	1	1.5	45	$73.90	$3,325.50
Estimated Annual Burden Total:				75	Estimated Annual Cost Total:	$5,542.50

Estimated Costs to Respondents

The cost information provided in Table 1 is based on the most current data available (August 2023) from the U.S. Bureau of Labor Statistics (BLS) Occupational Employment and Wage Statistics. For labor categories, the mean hourly wage for grant project directors/managers and community partner directors/managers (comparable BLS category is 11-9151 “Social and Community Service Managers”) of $38.13 was used, for evaluators (comparable BLS category is 19-3099 “Social Scientist and Related Workers”) a mean hourly wage of $45.46 was used, and for grant staff and community partner agency staff (comparable BLS category is 21-1021 “Child Welfare and School Social Workers”) a mean hourly wage of $27.25 was used. The average hourly wage across these respondent types is $36.95. To account for fringe benefits and overhead this rate was multiplied by two, which equals $73.90.

Labor categories and wage information were obtained from the following website:

https://www.bls.gov/oes/current/oes_stru.htm

13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

There are no additional costs to respondents.

14. Annualized Cost to the Federal Government

The estimated costs to the federal government for the data collection are indicated in Table 2. The total cost to the federal government for this collection is $22,500. These costs will be incurred within one year, as all data is expected to be collected over the course of about three months.  

Table 2. Estimated Annualized Costs to the Federal Government 

Cost Category	Estimated Costs
Data collection	$5,100
Data analysis	$17,400
Total costs over the request period	$22,500

JBA uses the General Services Administration rates for development of project budgets and used the rate of $168.11 (Project Director II) for the cost estimates. The estimates include the loaded costs and fees of study team staff time on administration of the surveys, conducting focus groups, and analyzing the collected data. Specifically, costs for data collection include study team time for programming the surveys into Qualtrics and administering them to invited respondents, scheduling and conducting focus groups, and follow-up activities related to transcribing focus group recordings. Costs for data analysis include study team time for conducting quantitative analysis of the survey data and qualitative coding and analysis of focus group data.  

15. Explanation for Program Changes or Adjustments

This is a new information collection.

16. Plans for Tabulation and Publication and Project Time Schedule

Data from this information collection will be summarized into a final report for CB that will provide information on grant interventions, implementation successes and challenges, factors that influenced implementation (e.g., collaboration with partners, the COVID-19 pandemic), achievement of outcomes of interest, including CFSR outcomes, and lessons learned. The final report and executive summary will be published on CB and JBA websites and disseminated to grant recipients and the public. The findings may also be shared during presentations and webinars with the public. There is no plan to make the raw, unanalyzed data collected available on the agency’s website, other federal government websites (e.g., data.gov) or in a restricted-access environment.

JBA plans to schedule and conduct the focus groups and administer the survey following OMB approval. Data collection will take place over about three months. Data analysis is then expected to take place within a month. Findings from the analysis of the survey and focus group data will be incorporated into the final report.

17. Reason(s) Display of OMB Expiration Date is Inappropriate

Does not apply.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

1 Note that the 60-day notice in the Federal Register included this in the title, but the 30-day notice has been revised.

2 OMB #: 0970-0406; ACF Performance Progress Report – Program Indicators