Instrument 4 Demonstration Guide_3_20_23_clean

Instrument 4

Child Welfare Study to Enhance Equity with Data (CW-SEED) Demonstration Guide

The demonstration guide will help the CW-SEED project team understand how and to what extent local child welfare agencies and other organizations involved with child welfare agencies collect and use quantitative and qualitative data to examine equity in child welfare services and family outcomes. The respondents will include staff from the child welfare agency, partner agencies, and community organizations. The observation guide will complement the interviews. Whenever possible, interviewers will ask staff to demonstrate how data practices work to support the agency or organization’s efforts to advance equity.

The average estimated public reporting burden for this collection of information is about 60 minutes per observation of the demonstration. Providing information is voluntary, and all responses that are collected are kept private to the extent permitted by law.

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid Office of Management and Budget (OMB) control number. The OMB number for this information collection is xxxx-xxxx, and the expiration date is xx/xx/20xx.

CW-SEED Demonstration Guide

Key definitions:

1. Data: For this conversation, we are defining data as information that is collected about individuals and families that come into contact with the child welfare system. Examples include information about age, gender identity, disability, and race/ethnicity, as well as descriptive information such as household structure, or the events that led to a child being placed in out-of-home care. In this study we are particularly interested in information agencies are collecting that can help assess and address equity, or inequities, in the child welfare system at the local level.

2. Data practices: We use the term data practices to broadly encompass all activities that involve data, including data planning, collection, access, and analysis; use of statistical tools and algorithms; and data reporting and dissemination. Unless otherwise specified, when thinking about data practices, please consider practices across the continuum of child welfare services, from prevention to addressing issues of child abuse and neglect, through permanency or other discharges (such as, aging out).

3. Data lifecycle: The data life cycle refers to the sequence of stages a particular unit of data goes through. In CW-SEED, we consider the data life cycle to include data planning, collection, data quality assessment, data organization, analysis, equity assessment, reporting, and dissemination. Feedback could then inform subsequent rounds of data planning, collection, and so on.

4. Equity: The consistent and systematic fair, just, and impartial treatment of all individuals, including individuals who belong to underserved communities that have been denied such treatment, such as Black, Latino, and Indigenous and Native American persons, Asian Americans and Pacific Islanders and other persons of color; members of religious minorities; lesbian, gay, bisexual, transgender, and queer (LGBTQ+) persons; persons with disabilities; persons who live in rural areas; and persons otherwise adversely affected by persistent poverty or inequality. (Consistent with Executive Order (EO) 13985 [Advancing Racial Equity and Support for Underserved Communities Through the Federal Government])

5. Underserved communities: Populations sharing a particular characteristic, as well as geographic communities, that have been systematically denied a full opportunity to participate in aspects of economic, social, and civic life. (Consistent with EO 13985 [Advancing Racial Equity and Support for Underserved Communities Through the Federal Government])

Brief Script

1. Introduce the moderator and co-facilitator

Thank you again for taking the time to demonstrate [name of data system/process/tool/analysis/reporting process] for us. My name is [NAME,] and my colleague is [NAME]. We are from Mathematica, an independent research firm, and we are here to learn about [DATA PRACTICE OF INTEREST].

2. Explain the project and purpose of discussion

I am going to start out by giving you a bit of background and talk about why we wanted to meet with you today. We are conducting the Child Welfare Study to Enhance Equity with Data (CW-SEED) project for the Office of Planning, Research, and Evaluation in the Administration for Children and Families at the U.S. Department of Health and Human Services. This project is designed to understand how data practices may be implemented by child welfare agencies to advance equity and address inequities. By data practices we mean the planning, collection, access, and analysis; use of statistical tools and algorithms; and data reporting and dissemination. Findings from this study are intended to identify emerging practices and lessons learned.

We are interested in hearing about the approaches, processes, challenges, and facilitators to using data practices to advance equity in your [AGENCY/ORGANIZATION]. We are interviewing leaders, supervisors, direct service and data staff from the child welfare agency, partner agencies, and community organizations about the data practices they are engaging in to advance equity. We are also speaking with members of advisory groups that work with [AGENCY/ORGANIZATION].

We have provided a copy of the Consent Form. I’m going to review the content of that form before we begin.

3. Privacy and recording [Read this section verbatim]

We expect this demonstration to take up no more than 60 minutes. Before we start, I want to let you know that your participation is voluntary. We will use the information we observe and you share with us to write a summary of what we have learned. We will not connect your name to any of your responses, so please feel free to talk openly about your opinions. We will keep your identity private to the extent permitted by law.

We will be taking notes, but we also want to record the conversation to make sure we capture the information you share accurately when we write reports. We will destroy the recording at the end of the project. If you want to say anything that you do not want recorded, please let me know, and I will be glad to pause the recorder and stop taking notes.

There are no consequences if you choose not to participate in this discussion. If you do not know the answer to a question, please say so, and we will simply move on. You do not have to answer any questions that you don’t want to answer.

We also ask that you keep the discussion private, and do not share what we discuss here with others outside this room.

If the demonstration will include case-level data we would prefer to see testing or training data, or deidentified data, if it is available. If such data is not available, we can comply with your [AGENCY/ORGANIZATION]’s nondisclosure requirements as needed.

Do you have any questions on the study?

Do you agree to participate in the study

Do we have your permission to record the conversation for notetaking purposes only?

I’ll start the recording. (start recorder only if participant(s) agree to be recorded).

Interviewer and note taker: Whenever possible we will observe demonstrations of data systems, processes, tools to collect data or information, and analysis and reporting processes related to the equity work being done by the child welfare agency and its partners. Use this demonstration guide to collect detailed descriptions of what you observe. The questions in the guide are separated by the type of demonstration being conducted. Repeat sections as needed if you are observing multiple data systems, processes, tools, analyses, or reports.

Type of demonstration	Description
Respondent background Only provide responses for this section if not completed in another interview
Job title
Length of time at agency or organization
Job responsibilities
Partner staff: Agency or organization’s relationship with child welfare agency
Data system
What data system is being observed?
What data elements related to equity are collected in, stored in, or reported from the data system? In particular, what data elements are available on demographic, socioeconomic, and other characteristics which can be used to assess equity among populations, such as race/ethnicity or SOGIE data? What other data elements are available, such as those on services or outcomes, that can be used to assess equity among populations?
Who is responsible for collecting and entering the data into the data system? Is this the same for all data, including data elements related to equity?
How are equity-related data elements entered? From a drop-down menu of options, as open text fields, and/or another way? Provide details including menu options if applicable.
Are data reviewed for accuracy? What are the processes for reviewing data for accuracy? What happens when an inconsistency is discovered?
Can the data fields related to equity in this data system be skipped, omitted, or left blank? What percentage of the fields related to equity are missing data?
Can data be updated in the system? What elements are updated? Under what circumstances? How is this done? Who can make updates? For example, can only the assigned caseworker make updates, or are there data quality specialists who address this? Is there an automated process in place to make updates? If data are updated, what happens to the previous entry? Are there dates associated with each new data update (e.g., SOGIE category ‘Q’ on mm/dd/yyyy; but then updated to SOGIE category ‘L’ on mm/dd/yyyy, etc.)?
Who has access to this data? Are the data shared? If so, with who? How is this done?
Are reports generated from the data? If so, what is the process for getting the report? Who has access to these reports? How are these reports used (such as, to reduce inequities in removals, to improve services)?
What resources (manuals/guides/codebooks) are available for this data system? What trainings, including onboarding, are available for this data system?
Tool or other resource for collecting data or information related to equity Tools and other resources include forms, policies, or practice guidance case workers use to collect intake information related to equity in a structured way.
What tool or resource is being observed?
What is the purpose of the tool or resource?
What data system is involved?
What data elements related to equity are involved?
What is the process for using this tool or resource? Describe in detail.
Who uses this tool or resource?
Is the tool or resource used at a specific point during the agency or organization’s contact with a family or child? Describe when it is used and under what circumstances.
What training, guidance, or policies exist to support staff in using the tool or resource?
Are there aspects of using the tool or resource that could be prone to error or bias? If so, describe. Describe any guardrails in place to reduce the likelihood of error or bias.
Can any data related to equity that is collected using this tool or resource be skipped, omitted, or left blank? What percentage of the fields related to equity are missing data?
What happens after the tool or resource has been used? Who is responsible for these next steps?
Is the resulting data or information shared? If so, with whom? How is this done?
Is the data or information used to generate reports? If so, how is this done?
Analysis and use of algorithms
What analysis (or use of algorithm(s)) is being observed or reviewed?
How is equity included in this analysis? For example, is the analysis conducted specifically to assess equity? Is the analysis for a broader purpose but equity is also considered?
What is the unit of analysis? For example, is the analysis being conducted on the individual, family, supervisor, office, or regional level?
What data are used? Describe in detail. Include information about how data are broken out by the different dimensions of equity (such as race/ethnicity, SOGIE), the level of detail shared, and which identities might be grouped together during reporting.
What is the process for conducting this analysis? Is the process for conducting the analysis automated?
Who conducts this analysis?
What is the purpose of this analysis (such as, to understand inequities to improve services)?
How often is this analysis conducted?
How easily could it be adapted to include other data element relevant to equity, use with other datasets, etc.?
What training, guidance, or policies exist to support staff in conducting the analysis?
What is the quality of the data? Provide examples. How frequently is the data needed for this analysis missing?
Are there parts of the analysis that could be prone to error or bias? If so, describe. Describe any processes in place to reduce the likelihood of error or bias.
Reporting
What report is being generated during this demonstration?
What is the purpose of the report? With whom is the report shared?
What is the process for generating this report? What data is used? Describe in detail. Include information about how data are broken out by the different dimensions of equity (such as race/ethnicity, SOGIE), the level of detail shared, and any identities that are grouped together.
Who generates this report?
How often is this report produced?
What training, guidance, or policies exist to support staff in generating the report?
What is the quality of the report? Provide examples. How frequently is data that is needed for the report missing?
Are there parts of the process of generating the report that could be prone to error or bias? If so, describe. Describe any guardrails in place to reduce the likelihood of error or bias.
Other data practices Use this section to record observations of demonstrations that do not fit into the above categories.
What data practice is being demonstrated?
What is the purpose of this data practice?
What data related to equity is involved (such as race/ethnicity or SOGIE data)?
Who is responsible for completing this data practice?
Are data reviewed for accuracy, missingness, or other aspect of quality? Can data fields related to equity be skipped, omitted, or left blank? What percentage of the fields related to equity are missing data?
Are data updated? Under what circumstances? How is this done? Who can make updates? For example, can only the assigned caseworker make updates, or are there data quality specialists who address this? Is there an automated process in place to make updates? If data are updated, what happens to the previous entry? Are there dates associated with each new data update (e.g., SOGIE category ‘Q’ on mm/dd/yyyy; but then updated to SOGIE category ‘L’ on mm/dd/yyyy, etc.)?
Who has access to the data involved in this data practice? Are the data shared? If so, with who? How is this done?
Are the data used to generate reports? If so, how is this done? Who has access to these reports? How are these reports used? How frequently are they generated?
Are there aspects of the data practice that could be prone to error or bias? If so, describe. Describe any guardrails in place to reduce the likelihood of error or bias.
General questions Respond to these questions for all observations
Has the data practice been revised or changed over time? If so, how has it changed? Why has it changed?
How has the data practice been used to understand, assess, and advance equity? How has the agency or partner organization used the information resulting from the data practice? How has this informed or shaped policy or practice (such as, to identify LGBTQ+ prospective foster or adoptive parents, to identify or interventions for specific demographics or client characteristics)?
How effective is this data practice? Has the data practice improved equity? For example, has the data practice helped to reduce disparities, provide equitable access to services, or help families achieve more equitable outcomes? Respond based on perception, respondent experience, or evidence. Is there evidence to support the data practice’s effectiveness? If so, describe.
What are the benefits and strengths of this data practice?
What are the challenges associated with this data practice?
How could this data practice be improved?
How easy or challenging would this data practice be to adapt to other jurisdictions? What would need to be in place for another jurisdiction to implement the practice?
Has the data practice changed over time? If so, how has it changed, why has it changed, and how did the agency or organization determine a change was needed?
Additional information
Use this section as needed to enter additional notes or contextual information
Additional comments