Supporting Statement A - BIAS NG Generic IC - Allegheny Child Welfare Site

ACF Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) Project

Generic Information Collection Request for Qualitative and Descriptive Quantitative Data Collection for a Child Welfare Site

0970-0502

SUPPORTING STATEMENT

PART A

June 2018

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

appendices

Appendix A: BIAS-NG Client Survey

Appendix B: BIAS-NG Semi-Structured Protocol for Client Interviews

Appendix C: BIAS-NG Staff Survey

Appendix D: BIAS-NG Semi-Structured Protocol for Staff Interviews

A1. Necessity for the Data Collection

The Office of Planning, Research, and Evaluation (OPRE) at the Administration for Children and Families (ACF), U.S. Department of Health and Human Services (HHS), seeks Office of Management and Budget (OMB) approval to conduct interviews and surveys with Child Welfare program administrators, staff, and clients in Allegheny County, PA to understand the mechanisms and effects of an intervention informed by behavioral science and intended to improve program outcomes. This information collection (IC) activity is planned as part of ACF’s Generic Clearance for the Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) project. The goal of the BIAS-NG Generic Clearance is to conduct qualitative and descriptive quantitative research to identify and understand the psychological and behavioral factors that can affect the effectiveness of human service programs in the areas of Child Welfare (CW) and TANF.

Study Background

The BIAS-NG project builds on a prior OPRE project, the Behavioral Interventions to Advance Self-Sufficiency (BIAS) project, which relied exclusively on administrative data to test the short-term impact of small “nudge” interventions in human services programs. Going beyond the work conducted for BIAS, the BIAS-NG project will test new interventions in more domains and collect a wider range of data.

The study described in this generic IC is launched in collaboration with the child welfare office in Allegheny County. The behaviorally informed intervention is designed to improve family engagement in their case planning process. In addition to administrative data on outcomes, we plan to collect administrative data on implementation of the intervention as well as qualitative and quantitative information from program staff and participants to better understand the mechanisms and effects of behavioral interventions.

Legal or Administrative Requirements that Necessitate the Collection

There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose of Survey and Data Collection Procedures

Overview of Purpose and Approach

• Context and Intervention

Allegheny County, PA’s Office of Children, Youth and Families (CYF) handles child welfare services. The Office is planning to try approaches to improving family engagement in the case planning process. BIAS-NG is assisting the County in evaluating the results of these innovations.

Specifically, the intervention is designed to improve families’ engagement in the first 60 days of having a case open with the Allegheny County child welfare agency, and specifically their participation in the creation of the family plan required by the County. Currently, the agency sends a letter to families notifying them of a case opening, the contents of which are dictated by law. The letter explains the appeal process, but does not explain steps the families need to take in the first 1-2 months of their case. Moreover, the letter is dense and may be confusing or intimidating to families. The County is trying a case-level intervention that provides two stages of outreach and support to families.

First, program group families will receive a flyer that accompanies their notice of case opening. The flyer uses graphics and simple language to explain what to expect in the first 1-2 months of their case. It lists case milestones and provides tips for navigating their case. It also contains clear information about who their caseworker is and how to get in touch with him or her, providing a quick and simple document to which families can easily refer. It prompts families to fill in the dates of their case milestones as they are scheduled. Overall, the flyer clearly indicates next steps, which the existing letter does not, and presents information in a simpler and friendlier manner than the existing letter.

Second, families with active cell phones will receive a sequence of weekly text messages that reinforce concepts in the flyer and prepare families for upcoming milestones. There are two text message sequences – one for cases in which a child has been removed from the home, and one for cases in which the child is still at home – to account for different timing of milestones in each instance. The text messages provide links to autodial or email their caseworkers; general reminders for case milestones; and helpful tips and information, including video links, for families to reference. The text sequence concludes around the time when families should receive their completed family plan from their caseworker and can begin to work on their action steps.

• Data Collection

This IC describes data collection related to phase 4 of the project described in the Generic Clearance.

We will collect information from staff (including supervisory staff) and clients for this IC. In the rest of this document and in Supporting Statement B, we include a description of:

• Planned qualitative data collection (see Appendices A-D for the specific instruments). Instruments include interview protocols and short surveys specific to each informant group (agency staff, and clients).

• Planned qualitative analyses. Audio recordings and notes from interviews, as well as open-ended responses on surveys, will be analyzed for patterns and themes.

• Administrative data that the agencies are already collecting and that the project will utilize.

Universe of Data Collection Efforts

Client survey (Appendix A): The research team will use client surveys to determine if program clients feel differently about their experience with the agency than control group clients do, as well as allow the research team to learn about general concerns they may have with agency processes. We plan to survey clients up to four times via text message. Clients will have the opportunity to opt-out of survey messages. These data will be used primarily for the implementation study, though information on whether program group clients felt more prepared than control group clients will also be used as a mediator for the impact study.

Client interviews (Appendix B): For the implementation study, the research team will interview program clients to determine whether the barriers identified during the exploration phase were addressed by the intervention materials, and to understand overall client experiences. In Allegheny County, we propose to interview 2-3 clients in each of the 6 offices in the study. These interviews will help the research team understand how well the intervention was implemented. Not all questions will be asked of each respondent, based on the participant’s background or experience. We will reduce burden by asking only relevant questions.

Staff/administrator survey (Appendix C): The research team will survey casework staff and case supervisors about their attitudes and perceptions of clients and their work, to help the team understand the context for the intervention. This survey will help us identify any cognitive biases staff may have about clients and their work. The research team will use this survey to inform the implementation study.

Staff interviews/focus groups (Appendix D): The research team will use these interviews to shed light on what aspects of the intervention worked well and which did not, and how the interventions changed client-staff interactions, if at all. If it is more convenient for respondents, these discussions may be held in group settings. Each group discussion will include staff at the same or similar levels. For example, one group discussion may be held with multiple front-line workers, such as case workers or outreach specialists. A separate group discussion may be held with supervisors of front-line staff. These data will also contribute to the implementation study.

In addition to collecting data from staff and clients with interviews, and surveys, we also intend to supplement this information with administrative data the agencies are already collecting. Collecting administrative data on case outcomes for the impact evaluation will not impose a burden on respondents or record keepers, as we intend to ask sites to provide data as it currently exists. We will not be requesting that it be provided in any particular format that is different from the format in which the agency typically keeps it (see Supporting Statement B). In addition, we will not ask more than nine individuals to provide the administrative data.

Research Questions and Study Design

Table 1 presents research questions that will be addressed by information collected in this phase.

Impact Research Question:

What is the effect of offering families supplemental orientation information at case opening and weekly text message reminders and supports, on top of a notification letter and ad-hoc communication from caseworkers, on completing milestones in the first 1-2 months of their case?

Impact Study Design:

We will conduct a randomized field trial for newly opened cases. To maximize our sample size and set a positive trajectory for families, we will randomly assign all cases accepted for service on the day of the case opening. This will essentially start at the beginning of the case flow into child welfare services, and hopefully avoid sample attrition problems. We will use a standard 50/50 randomization and block by the six regional offices, to control for baseline differences in case practice and case inflow rates between offices. A case assigned to intervention could involve multiple adults.

Implementation Research Questions:

1. To what extent were the interventions implemented with fidelity?

2. What challenges and barriers did the site experience with implementation?

3. What is the organizational structure and culture, and how does it support or hinder responses to the behavioral intervention?

4. What are participant perspectives on the agency, staff, and the intervention?

5. What are staff perspectives on their work for the agency, clients and client response to the intervention? What cognitive biases may be in play?

Implementation Study Design:

We will conduct an implementation study to describe and document the intervention, how it operated, and provide information about the contrast in treatment between the research groups—both whether the planned contrast between the treatment and the control condition occurred (implementation fidelity) as well as how the treatment implemented actually differed from the status quo (treatment contrast). This information will be important for interpreting the findings of the impact study.

We will gather information to answer these questions through interviews with clients; either interviews or focus groups with staff; and surveys with clients and staff. The qualitative data collection activities are essential to conducting implementation research. Please see Appendices B-E for interview and survey questions.

Table 1: Research Question and Instrument Matrix

Research Questions	Administrative data/ MIS	Client survey	Client interviews groups	Staff survey	Staff interviews/ focus groups
What is the effect of the intervention on case outcomes?	X
To what extent were the interventions implemented with fidelity?	X	X	X	X	X
What challenges and barriers did the site experience with implementation?				X	X
What is the organizational structure and culture, and how does it support or hinder responses to the behavioral intervention?				X	X
What are participant perspectives on the agency, staff, and the intervention?		X	X
What are staff perspectives on their work for the agency, clients and client response to the intervention? What cognitive biases may be in play?				X	X

A3. Improved Information Technology to Reduce Burden

We will use conference calls and emails to the extent possible to minimize burden on staff. If a phone or video interview is easier for caseworkers' schedules, we will conduct interviews in that mode.

Surveys for staff will be administered on the web and emailed directly to them.

Surveys for clients will be administered via text message, to minimize the need for clients to have a computer and internet connection.

A4. Efforts to Identify Duplication

We have worked carefully with the agency to understand the data the agency routinely collects on families in its caseload. None of the data currently collected by the agency would allow us to assess whether or not our intervention (the introductory flyer and the planned text messages to families) improves client understanding of the case process and/or their willingness to engage in the process. In addition, the study team will not collect information from the child welfare agencies and staff that is available from existing sources.

A5. Involvement of Small Organizations

We do not anticipate any small organizations to be affected by these information collections. Nonetheless, we will schedule interviews at times that are convenient to participants in order to minimize disruption of daily activities.

A6. Consequences of Less Frequent Data Collection

Rigorous evaluation of innovative initiatives is crucial to building evidence of what works and how best to allocate scarce government resources. Not collecting information about the implementation and effect of the intervention would hinder the government’s ability to learn how interventions were implemented and why and to what degree the interventions achieved the outcome desired.

A7. Special Circumstances

There are no special circumstances for this data collection.

A8. Federal Register Notice and Consultation

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on May 23, 2017, Volume 82, Number 98, page 23572, and provided a 60-day period for public comment. No substantive comments were received during the notice and comment period.

The research team consulted with behavioral science experts on the intervention design and on select items in the data collection protocols.

A9. Incentives for Respondents

We propose for clients participating in interviews to receive a gift card of $20, which is standard for clients in Allegheny County who have participated in other data collection activities with the County. Our hope is that a gift card will help offset the financial burden that may result from extenuating travel, additional cell-phone data or phone minutes, or child care costs associated with interviews. Additionally, families involved with the Office of Children, Youth and Families (CYF) are likely to be under large amounts of stress and potentially difficult to reach, especially given that they may have an unfavorable view of CYF. We expect a $20 gift card will help to offset these incidental costs associated with participation and encourage families who may be stressed or unhappy with CYF to participate. We do not believe the incentive is so high as to be coercive for clients.

Incentives will not be used as a substitute for other best-practice persuasion strategies designed to increase participation, such as explanatory advance letters, endorsements by people or organizations important to the population being surveyed, and assurances of privacy.

We have secured Institutional Review Boards (IRB) approval for the study, including use and monetary value of the use of incentives for interviews with clients.

A10. Privacy of Respondents

All respondents who participate in research under this clearance will read or be read a statement that will explain the study and will inform individuals that their participation is voluntary and of the extent of their privacy as respondents (informed consents are included in each instrument in Appendices B-D). Participants will be told verbally that their conversations will not be shared in a form that identifies them with anyone outside the research team. As ACF’s prime contractor, MDRC plans to implement all data collection activities. Information will be kept private to the extent permitted by law and in accordance with current federal information security standards and other applicable regulations.

MDRC employees are required to maintain and process quantitative and qualitative data in designated project folders on the MDRC network. With the exception of the temporary storage of data during onsite collection, MDRC employees are not allowed to download, keep, or process individual-level data on the hard drives of their MDRC work stations or any other storage. Information will not be maintained in a paper or electronic system from which they are actually or directly retrieved by an individuals’ personal identifier.

The project Data Manager will organize BIAS-NG project folders and will supervise storage of BIAS-NG data files on a “need-to-know” basis. Following standard MDRC practice, the project Data Manager and project programmers will replace all personally identifiable information (PII) from incoming source data with a randomly generated project ID number. Also these files will be saved in secure folders with limited access on a “need-to-know” basis. Thereafter, analysis will be performed on data files stripped of PII (we will initially retain PII in order to merge individual data sources to create an analysis file). All reports, tables, and printed materials are limited to presentation of aggregate numbers. MDRC will destroy all paper records and electronic records containing PII when no longer needed for research purposes in accordance with funder and contractual requirements, as well as MDRC retention policies.

MDRC’s IRB has approved this study.

A11. Sensitive Questions

We are asking some sensitive questions in this data collection in terms of client relationships with staff and vice versa. For example, we ask clients in a survey to rate the information and preparation they received from their caseworker. And we ask staff in a survey to indicate their perceptions of obstacles facing their clients. These answers will help the study team identify barriers to and facilitators of engagement. We assure clients that their caseworkers will not see their responses, and vice versa, to encourage honest responses.

A12. Estimation of Information Collection Burden

Table 2 provides details about how this estimate of burden hours and costs were calculated.

We expect to speak with 2-3 clients, and 2-3 staff, in each of six offices in Allegheny County, for a total of 12-18 clients and 12-18 staff. We anticipate extending the staff survey to 180 casework staff and supervisors, with an expected response rate of 60%, or 108 respondents. We anticipate extending the client web survey to 1,500 clients, with an expected response rate of 40%, or 600 respondents. The estimate below represents an upper bound on potential burden.

We calculated the overall burden per respondent by multiplying the frequency of response by the time to complete each data collection item among respondents. We anticipate that focus groups or interviews for staff and clients will each take 1 hour to complete. We anticipate the staff surveys to take approximately 15 minutes to complete online, and the client survey to take 8 minutes total (2 minutes at each of 4 occasions maximum) to complete.

Table 2: Burden Hours for Allegheny County

Instrument	Total Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Total Burden Hours	Average Hourly Wage	Total Cost
Client survey (Appendix A)	600	4	.04	96	$15.55	$1,492.80
Client interviews (Appendix B)	18	1	1	18	$15.55	$279.90
Staff survey (Appendix C)	108	1	.25	27	$33.90	$915.30
Staff interviews/focus groups (Appendix D)	18	1	1	18	$33.90	$610.20
Total	744			159		$3,298.20

Total Cost

To compute the total estimated cost for staff, the total burden hours were multiplied by $33.90, the mean hourly earnings for management, professional and related workers in the civilian workforce as reported by BLS NCS (2010).¹ To compute the total estimated annual cost for clients, the total burden hours were multiplied by $15.50, the mean hourly earnings for high school graduates reported by BLS NCS (2010). The estimated total cost is $3,298.20.

A13. Cost Burden to Respondents or Record Keepers

There are no additional costs to respondents.

A14. Estimate of Cost to the Federal Government

The total cost for the data collection, analysis and reporting activities under this current request will be approximately $400,000.

A15. Change in Burden.

This is an individual Generic IC under the BIAS-NG Generic Clearance (0970-0502).

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

Phase 4: Evaluation: Phase 4 consists of implementing the behavioral intervention(s) and evaluating them, and collecting long-term outcomes.

Random assignment will begin immediately following OMB approval. Data collection will take place at different times following OMB approval. Implementation data from administrative data and client surveys will begin 1 month following approval. Implementation data from interviews will take place 6 months following OMB approval. Analysis will begin 9 months after OMB approval.

Phase 5: Dissemination: Dissemination efforts during the time of this clearance includes site specific reports, infographics, dissemination products aimed at practitioners, sharing findings at conferences, and publicizing our findings and our work on social media. Dissemination efforts are expected to begin after analysis concludes (about 18 months after OMB approval).

A17. Reasons Not to Display OMB Expiration Date

All instruments will display the expiration date for OMB approval.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

1 U. S. Department of Labor, Bureau of Labor Statistics “National Compensation Survey: Table 1: Summary Mean hourly earnings and weekly hours for selected workers and establishment characteristics.” 2010. http://www.bls.gov/ncs/ocs/sp/nctb1344.pdf.

10