Evaluation of Resources to Support the Identification and Care of Children with Prenatal Substance or Alcohol Exposure in the Child Welfare System
OMB Information Collection Request
0970 – 0608
Supporting Statement Part A – Justification
October 2023
Submitted By:
Children’s Bureau
Administration for Children and Families
U.S. Department of Health and Human Services
Circumstances Making the Collection of Information Necessary
The Children’s Bureau (CB) in the Administration on Children, Youth and Families (ACYF), Administration for Children and Families (ACF), U.S. Department of Health and Human Services (HHS) is partnering with the Centers for Disease Control and Prevention to support child welfare agency staff in the identification and support of children in the child welfare system living with prenatal substance exposure (PSE), including prenatal alcohol exposure (PAE). To this end, we are developing a toolkit of resources that child welfare agencies can use to:
Increase awareness, understanding, and knowledge of PAE/PSE; and
Plan and implement internal and cross-system processes, in partnership with key stakeholders, that help identify, assess/evaluate/screen, share information about, and provide care and support to children with prenatal exposure and their families.
The CB is undertaking a multi-stage evaluation designed to understand the extent to which the toolkit is achieving these intended outcomes. The following activities will contribute to the evaluation: (1) usability testing and (2) a formative evaluation. The second stage of the evaluation builds on the first stage. Usability testing of the toolkit was conducted November 2022-March 2023 (under Formative Generic for Program Support OMB #0970-0531, exp 9/30/25) to gather feedback from child welfare staff in the field to determine the usability of toolkit modules before the toolkit is finalized. Based upon usability testing findings, an equity review was conducted and the toolkit has undergone modifications based upon the findings. Data collection for the formative evaluation was approved in April 2023 (OMB# 0970-0608, exp 4/30/26).
This current information collection request is a modification to the formative evaluation data collection activities. Specifically, we propose to:
Add a focus group to assess how toolkit users are applying the modified resources in practice and to understand if and how the modifications to toolkit resources related to diversity, equity, and inclusion are leading to intended supports for children and families.
Implement minor updates to survey instrument items to better align with the modified toolkit.
This information collection is authorized by the Child Abuse Prevention and Treatment Act Reauthorization Act, 42 U.S.C. § 5105 (2010). This information collection complies with the statutory requirement to carry out research designed to provide information needed to improve the well-being of victims of child abuse or neglect. Specifically, this information collection complies with the requirement for evaluation of practices and programs to improve activities such as identification, screening, medical diagnosis, forensic diagnosis, health evaluations, and services, including activities that promote collaboration between (1) the child protective service system; and (2) (i) the medical community, including providers of mental health and developmental disability services; and (ii) providers of early childhood intervention services and special education for children who have been victims of child abuse or neglect.
Purpose and Use of the Information Collection
The current information collection request – a formative evaluation of the toolkit – is the second stage of the two-stage evaluation which follows an initial stage for usability testing and incorporation of modifications to the toolkit. Formative evaluation is an evaluation of a program or product during early implementation to inform quality improvement and assess whether the program/product is ready for rigorous summative evaluation. The formative evaluation of the toolkit will differ from the usability testing process in that the formative evaluation (1) will be conducted once the toolkit is past the design phase and ready for implementation; and (2) is intended to assess actual implementation and short-term outcomes.
Findings from the formative evaluation of the toolkit will help to inform decision making about whether any revisions to the toolkit content or its implementation processes need to be made, to ensure that the toolkit functions as intended for the child welfare field, prior to any future phases of evaluation. The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information. Findings from the formative evaluation will be presented in a brief report that also outlines recommended revisions to the toolkit to be submitted to CB at the completion of the formative evaluation. A description of the formative evaluation, key findings, and how findings were applied in toolkit revisions may be included in the project final report and a capstone report that will in part outline considerations to the government regarding action steps for dissemination of the toolkit. The study team and CB may share key findings from the formative evaluation at conferences or in journal articles as part of an effort to disseminate information about the development and evaluation of the toolkit.
Guiding Research Questions
The formative evaluation of the toolkit will be guided by three research questions: (1) To what degree do agency staff find toolkit resources to be relevant and applicable to their work?; (2) To what degree do toolkit resources change agency staff attitudes and increase staff knowledge?; and (3) What implementation approaches and organizational supports facilitate toolkit use by child welfare agencies?
Study Design
To evaluate the toolkit, the study team will work with selected child welfare agency sites in up to two states (see B-1). The staff at the sites will implement the toolkit modules with implementation support from the study team over the course of 6 or more months. The team will collect survey data from the toolkit users at multiple time points to measure users’ reactions to the toolkit; potential changes in their attitudes toward and knowledge about PAE/PSE-related issues; and potential changes in their perceived competency and potential to transfer their newly acquired knowledge (if any) to their work in child welfare. The team will conduct one focus group at each site to assess application of the toolkit in their casework, whether processes and resources in the toolkit are achieving intended applicability for children and families from diverse contexts and cultures, and barriers and facilitators in applying the toolkit. These focus groups are the focus of this revision request.
Data Sources
The data sources for this effort include five surveys and one focus group (see Table A-1). The surveys were approved by OMB in April 2023; this revision request is to add the focus groups to the data collection and provide updates to survey items based upon usability findings. All data will be collected over the course of 5-6 months in 2024.
1. Survey of Reactions to the Toolkit. This is a 13-item, web-based survey designed to provide data to help CB understand how toolkit users (i.e., child welfare agency staff, see table B-1 in SS-B) perceive the usefulness of the toolkit. The survey will be administered to child welfare staff at the end of their site’s toolkit review process.
2. Survey of Attitudes. This is a 27-item, web-based survey designed to provide data to help CB understand how toolkit users feel about parental substance use, identification of parental substance use and PSE, and identifying and caring for prenatally exposed children. The survey will be administered to child welfare staff at baseline (i.e., prior to their introduction to the toolkit) and again at the end of their site’s toolkit review process.
3. Survey of PAE/PSE-Related Knowledge. This is a 33-item, web-based survey designed to provide data to help CB understand what toolkit users learn and retain about aspects of identifying parental substance use and PSE and identifying and caring for prenatally exposed children. The survey will be administered to child welfare staff at three time points: at baseline, directly after their review of a given module of the toolkit, and again 6-8 weeks after the end of their site’s toolkit review process.
4. Survey of Transfer Potential and Perceived Competency. This is a 37-item, web-based survey designed to provide data to help CB understand the extent to which toolkit users may go on to apply newly acquired knowledge and skills to their work. The survey will be administered to child welfare staff at the end of their site’s toolkit review process
5. Module-specific items from the Survey of Transfer Potential and Perceived Competency. Select items from the Survey of Transfer Potential and Perceived Competency (described above) that are specific to key toolkit modules will be administered to child welfare staff directly after their review of those modules, to capture their immediate perceptions about the extent to which they may go on to apply the newly acquired knowledge and skills to their work.
6. Focus groups on Enhancements to Practice after Implementation of Toolkit. Ten open-ended questions will assess toolkit user’s perspectives on changes to practice as a result of implementation and fit and alignment of the toolkit with the agency’s diversity, equity, and inclusion efforts and the diversity of the population of children and families they serve.
Table A-1. Instrument Description and Administration Details
Use of Improved Information Technology and Burden Reduction
Wherever possible and appropriate, information technology will be used to capture information and reduce burden relative to alternative methods of data collection. Administration of the surveys described above will be web-based, utilizing email notification and Qualtrics, a web-based survey technology. This will create efficiencies for survey administration, allowing flexibility and convenience for recipients, and ideally result in a user-friendly experience for respondents. Administration of the focus groups will be through collaborative video conferencing technology and scheduled at a time convenient for informants. With the permission of informants, focus groups will be audio recorded and transcribed to maximize detailed and accurate notes and to minimize the need to go back to informants to clarify what was said.
At each designated data collection timepoint (described above), respondents will receive an email notification from the study team, inviting them to complete one or more of the survey instruments and/or to join the focus group. The survey invitation email will include a web-link to the online survey in Qualtrics. Targeted respondents are expected to be able to access the online surveys, though a hard copy of the surveys can be provided to those who cannot access the surveys online.
Efforts to Identify Duplication and Use of Similar Information
The proposed information collection represents a unique source of data that are not already available. The toolkit is an innovative intervention that is still under development; as such, there are no existing sources of data that could be used to understand the short-term outcomes of this specific resource.
Impact on Small Businesses or Other Small Entities
No small businesses will be involved with this information collection.
Consequences of Collecting the Information Less Frequently
This study involves the systematic collection of data from toolkit users over the course of several months to inform quality improvement of the toolkit and assess whether the toolkit intervention is ready for rigorous summative evaluation. Less frequent data collection (e.g., collecting data at one time point instead of two or three [see table A-1]) would reduce the utility of the formative evaluation process in that it would not allow us to examine pre-post comparisons of attitudes and knowledge, which are key intended short-term outcomes of toolkit use. Understanding whether key short-term outcomes are trending in the right direction is one of the primary purposes of formative evaluation and a necessary precursor to any future evaluations.
Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
There are two aspects of the planned information collection that should be addressed as it pertains to the guidelines of 5 CFR 1320. First, the design of the study necessitates collection of survey data from toolkit users prior to and again after their exposure to the toolkit, to measure potential changes in their knowledge about and attitudes toward PAE-related issues. The interval between data collection time periods will in most cases be less than three months because in most cases it will not take toolkit users more than a month or so to become familiar with and begin to apply a given toolkit module. Thus, respondents may be asked to report information more often than quarterly. Second, in order to complete the study within 6 months (as described above), when we send links to the online surveys to toolkit users (as described above) we will typically ask that they complete the survey within two weeks (i.e., in fewer than the 30 days outlined in 5 CFR 1320.5). This not only keeps data collection moving forward, but also ensures that respondents are providing information about their reactions, attitudes, and knowledge immediately prior to or after exposure to the toolkit, which are the specific points that are the focus of the measurement.
Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on September 22, 2022, Volume 87, Number 183, page 57902, and provided a sixty-day period for public comment. During the notice and comment period, we did not receive comments. Child welfare evaluation consultants from University of Louisville School of Social Work and Family Science (see B-5) and federal staff reviewed and offered comments on the data collection and instruments.
ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this modification of the previously approved information collection activity (OMB #0970-0608). The notice was published on July 25, 2023, Volume 88, Number 141, pages 47883-84, and provided a sixty-day period for public comment. During the notice and comment period, we did not receive comments.
Explanation of Any Payment or Gift to Respondents
No payments or gifts will be provided to respondents.
Assurance of Confidentiality Provided to Respondents
Respondents will be informed of all planned uses of the data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. No assurance of confidentiality will be provided to respondents. The study team will not disclose any individual-level survey or focus group information to persons outside the study team. Information will not be published that could be used to identify individual respondents or participating agencies. All survey and focus group results will be analyzed and reported in the aggregate for the reports (i.e., brief report, project final report, and capstone report), conference presentations, and journal articles discussed in A-2.
All study team members will sign a study Data Security and Protection of Confidentiality agreement and any required site agreements.1 All of the study materials including all instruments and consent documents will be reviewed and approved by an Institutional Review Board (IRB) prior to initiation of the study. If required, all study materials will be submitted and approved by agency and state IRBs.
The evaluation will be guided by a written data security plan that articulates how the study team will ensure that all sensitive information in its possession is stored and maintained in accordance with ACF and Federal Information Security Management Act (FISMA) requirements. The study team will use an encrypted folder designated for this project in a Microsoft Office 365 SharePoint Site. The data security plan will document how all data collected will be desensitized and stored.
Justification for Sensitive Questions
One instrument in this proposed data collection effort, the Survey of Attitudes (see Instrument 2), is designed to collect information on toolkit users’ attitudes toward issues related to parental substance use identification of parental substance use/prenatal substance exposure, and caring for children prenatally exposed to substances. These issues can be considered sensitive and toolkit users (i.e., child welfare staff) may typically prefer to keep their attitudes toward these issues private. After usability data analysis and subsequent toolkit modification, some items were removed, to reduce the number of potentially sensitive items, and to enhance alignment with toolkit content that is intended to help change attitudes.
Collecting information about toolkit users’ attitudes toward these issues is necessary because it enables the study team to more fully understand whether and why toolkit users learn and apply the information in the toolkit. Child welfare staff attitudes matter because they can influence practice by impacting workers’ assessments of given situations and the decisions to be made (Sanbonmatsu et al., 2005). Child welfare staff may hold beliefs about whether identification of PSE is the role of child welfare or is likely to be beneficial to child processes, based on their direct experience or what they have learned on the job. These attitudes are likely to be important to acceptance and application of information in the toolkit about the identification and care of children with PSE. Training/learning in child welfare – particularly when learning is supported by coaching – may help decrease resistance and improve attitudes, beliefs, and acceptance of new practices (Allen, Hafter & Brook, 2020; Hatton-Bowers et al., 2015). Thus, it is hypothesized that use of the toolkit may result in positive changes in staff attitudes toward identification of PSE and, ultimately, transfer of learning.
The information collected about child welfare staff attitudes toward these issues will be used in analyses to (a) determine whether exposure to the toolkit contents changes attitudes; and (b) to understand whether child welfare staff attitudes serve as a potential obstacle or facilitator to acceptance of and application of the toolkit within a given study site. The data collected by the Survey of Attitudes will be kept private and will not be shared with toolkit users’ colleagues, supervisors, agency director, or external partners. The explanation of the purpose and use of the survey and the language to obtain consent are shown at the top of the survey instrument (see attached instruments). Respondents will have the ability to skip any survey questions they do not feel comfortable answering and to exit the survey without completing it at any point.
Estimates of Annualized Burden Hours and Costs
Estimated Burden Hours for Respondents
The estimates of burden hours for survey respondents shown in table A-2 were obtained through pilot testing of the instruments by staff of the contracted evaluation firms (see B-5). Three staff from James Bell Associates and one from ICF who have expertise in child welfare but were not involved in the development of the instruments took the surveys and made notes of the time (in minutes) that was required to do so. The average burden hour estimates shown below represent the mean number of minutes by instrument across the four pilot testers. The estimated number of respondents assumes approximately 8 respondents at each of the four sites for up to 32 respondents to the surveys.
The estimates of burden hours for the focus group shown in table A-2 were determined based upon the proposed 90-minute administration period. Based upon the usability phase of evaluation, similar focus group protocols of between 10-12 questions conducted with 6-8 informants were successfully completed in a 90-minute administration period. Seven of the 8 respondents at each site will be recruited to participate in the focus groups (all respondents except for the Director) for a total of 28 respondents.
Estimated Costs to Respondents
After applying hourly wage estimates to burden hours in each respondent category, the current annual cost to the respondents is $6413.99 (see table A-2). This cost information is based on the most current data available (May 2021) from the U.S. Bureau of Labor Statistics (BLS) Occupational Employment and Wage Statistics. For labor categories, the mean hourly wage for child welfare specialists (comparable BLS category is 15-1242 “Database Administrators”) of $49.29 was used, and for other child welfare staff (comparable BLS category is 21-1021 “Child, Family, and School Social Workers”) a rate of $27.25 was used. We estimated 4 specialists (including supervisors) and 4 other child welfare staff per site, with 4 sites participating in the study (for a total of 32 participants). The average hourly wage for the 16 specialists and 16 other child welfare staff is $38.27. To account for fringe benefits and overhead this rate was multiplied by two, which is $76.54.
Labor categories and wage information were obtained from the following website: https://www.bls.gov/oes/current/oes_stru.htm#21-0000
Table A-2. Estimated Annualized Burden Hours and Cost
Information Collection Title |
Total Number of Respondents |
Total Number of Responses Per Respondent |
Average Burden Hours Per Response |
Total/ Annual Burden Hours |
Average Hourly Wage |
Total Cost |
Survey of reactions to the toolkit |
32 |
1 |
.05 |
1.6 |
$76.54 |
$122.46 |
Survey of attitudes |
32 |
2 |
.17 |
10.88 |
$76.54 |
$832.76 |
Survey of PAE-related knowledge |
32 |
3 |
.27 |
25.92 |
$76.54 |
$1887.49 |
Survey of transfer potential and perceived competency |
32 |
1 |
.09 |
2.88 |
$76.54 |
$220.44 |
Module-specific transfer potential and perceived competency items |
32 |
5 |
.03 |
4.80 |
$76.54 |
$367.39 |
Focus groups |
28 |
1 |
1.5 |
42 |
$76.54 |
$3214.68 |
|
|
|
|
88.08 |
Estimated Cost Total |
$6741.64 |
Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers
There are no other costs to respondents and record keepers.
Annualized Cost to the Federal Government
The estimated costs to the federal government for the data collection are indicated in table A-3. The total cost to the federal government for this collection is $34,321.95. These costs will be incurred within one year, as all data will be collected over the course of 6–9 months in 2024.
Table A-3. Estimated Annualized Costs to the Federal Government
Cost Category |
Estimated Costs |
Survey administration |
$2887.11 |
Focus group administration |
$679.32 |
Data analysis |
$30755.52 |
Total costs over the request period |
$34,321.95 |
The estimates include the loaded costs and fees of study team staff time on administration of the surveys and focus groups and analysis of collected data. Specifically, costs for survey administration include study team time for programming the instruments into Qualtrics and administering them to invited respondents. Costs for focus group administration include study team time for scheduling and facilitation of the focus groups. Costs for data analysis include study team time for conducting quantitative and qualitative analysis of the collected data.
To estimate these costs, an average hourly wage of $56.61 was used for the study team staff (comparable to BLS category 19-3099 “Social Scientists and Related Workers”) with a multiplier applied to account for overhead.
Explanation for Program Changes or Adjustments
This is a request to add an additional information collection to this study. Specifically, we have added focus groups and estimate conducting these discussions with up to 28 individuals.
Plans for Tabulation and Publication and Project Time Schedule
Frequency and proportion distributions will be calculated to generate summaries and to examine variability in the survey data. Cross-tabulations and significance tests will be conducted as appropriate. SAS will be used for the quantitative analysis. Analyses will be conducted to determine subgroup variation, as appropriate.
The findings will be summarized in a brief report shortly after the conclusion of the formative evaluation to be submitted to the CB in July 2024; this report is intended as an internal document. It is anticipated (based on OMB approval) that the information collection activities will be administered and analyzed between January 2024 and May 2024 and report development will occur between June and July 2024. A description of the formative evaluation, key findings from the brief report, and how findings were applied to complete revisions to the toolkit will also be included as a part of the project final report (due to CB by September 2024) and may be included in the capstone contractual report that will in part outline considerations to the government regarding action steps for dissemination of the toolkit (due to CB by September 2024).
Reason(s) Display of OMB Expiration Date is Inappropriate
The OMB expiration date for the information collection and the OMB control number will appear on the instruments (see attached). A statement will appear that describes the public reporting burden and explains that an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.
Exceptions to Certification for Paperwork Reduction Act Submissions
No exceptions are necessary for this information collection.
Attachments
Appendix A: Site selection and recruitment communications
Appendix B: Description of project
Instruments
Instrument 1: Survey of reactions to the toolkit
Instrument 2: Survey of attitudes
Instrument 3: Survey of PAE/PSE-related knowledge
Instrument 4: Survey of transfer potential and perceived competency
Instrument 5: Module-specific transfer potential and perceived competency items
Instrument 6: Focus group protocol
References
Allen, A., Hafer, N., & Brooks, S. (2020). Understanding the role of coaching in implementing and sustaining interventions in child welfare: A review of the literature. Child Welfare, 98(2).
Hatton-Bowers, H., Pecora, P. J., Johnson, K., Brooks, S., & Schindell, M. (2015). Evaluating training to promote critical thinking skills for determining children’s safety. Journal of Social Work Education, 51, 298-314.
Sanbonmatsu, D. M., Prince, K. C., Vanous, S., & Posavac, S. S. (2005). The multiple roles of attitudes in decision making. In T. Betsch & S. Haberstroh (Eds.), The routines of decision making (pp. 101-116). Mahwah, NJ: Lawrence Erlbaum Associates Publishers.
1 The CB and CDC federal project officers are considered part of the immediate study team and therefore will have access to confidential information from participants and held to all of the data security standards.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Jones, Molly (ACF) |
| File Modified | 0000-00-00 |
| File Created | 2023-10-16 |