Supporting Statement A - BIAS NG Washington State TANF Site

Status of Study: This data collection is part of the Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) project. This new data collection pertains to Phase 4 (Evaluation) for the Washington State site within the Temporary Assistance for Needy Families (TANF) domain. More information on the context of the BIAS-NG project can be found in Section A.1.1 – Study Background.
Evaluation and Measurement. This data collection is intended to inform the understanding of the implementation of an intervention being evaluated by a randomized controlled trial (RCT) in this phase of the BIAS-NG project. This data collection includes interview and focus group protocols that will collect information about the treatment and control conditions. The data collection also includes qualitative research protocols to collect practitioner feedback on the intervention materials.
Type of Study. The study features an implementation study (described here) and an impact study consisting of two RCTs to assess the impact of behaviorally informed materials on attendance at TANF meetings and activities.
Utility of the information collection. This research is motivated by the low attendance rates of TANF participants at required meetings and activities in Washington State. This implementation research aims to provide context to help understand the results from the RCT. The results of the study and lessons learned from the design process can be shared with other practitioners in the TANF field and beyond to improve communication and engagement with social services clients. In addition, the RCT will be structured to build knowledge in the behavioral science field about the effectiveness of alternative communication approaches at spurring individual choices.

A1. Necessity for the Data Collection

The Office of Planning, Research, and Evaluation (OPRE) at the Administration for Children and Families (ACF), U.S. Department of Health and Human Services (HHS), seeks Office of Management and Budget (OMB) approval to conduct interviews with TANF program administrators, staff, and clients in Washington State to understand the mechanisms and effects of an intervention informed by behavioral science and intended to improve program outcomes. This information collection (IC) activity is planned as part of ACF’s Generic Clearance for the Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) project. The goal of the BIAS-NG Generic Clearance is to conduct qualitative and descriptive quantitative research to identify and understand the psychological and behavioral factors that can affect the effectiveness of human service programs in the areas of Child Welfare (CW) and TANF.

Study Background

The BIAS-NG project builds on a prior OPRE project, the Behavioral Interventions to Advance Self-Sufficiency (BIAS) project, which relied exclusively on administrative data to test the short-term impact of small “nudge” interventions in human services programs. Going beyond the work conducted for BIAS, the BIAS-NG project will test new interventions in more domains and collect a wider range of data.

The study described in this generic information collection request (ICR) is launched in collaboration with the Washington State Department of Social and Health Services (DSHS). The behaviorally informed intervention is designed to improve engagement of TANF clients in certain mandated employment services. In addition to administrative data on outcomes, we plan to collect administrative data on implementation of the intervention as well as qualitative and quantitative information from program staff and participants to better understand the mechanisms and effects of behavioral interventions.

Legal or Administrative Requirements that Necessitate the Collection

There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose of Survey and Data Collection Procedures

Overview of Purpose and Approach

Context and Intervention

This Generic IC pertains to Washington State, where the Department of Social and Health Services (DSHS) handles TANF program services. This study is focused on the DSHS’s goal of improving client engagement in employment services under the TANF program.

Participation in TANF WorkFirst (WF) activities is required by the state for clients to receive full TANF benefits. We have identified a number of issues with the current form and process that case managers use to document participants’ participation plan:

The form is dense.
The content highlights allowable reasons for non-participation and the benefit sanctions process if clients fail to participate for non-allowable reasons.
The steps participants should follow and contact information for employment services case managers are buried in this form, the content of which is automatically filled with notes case managers type into their electronic case files.

This form and process limit client engagement in making their participation plan. To address these issues, we designed a bundle of supplemental print materials to be collaboratively used by case managers and participants during the intake session. The materials are designed to 1) clarify participation expectations, 2) focus on reasons for participation, and 3) help participants make a plan to take the next step required to participate (e.g., attend an orientation session at their WF activity provider). We are testing a one-stage, household-level intervention that randomly assigns new TANF applicants across five field offices to either a) the program group, where they complete their Comprehensive Evaluation (CE)¹ with staff who are trained to use the intervention materials throughout the intake session, or b) the control group, where they will receive business-as-usual services.

There are three components in the bundle of intervention materials:

First, a set of flashcards explain the different WF activity options in a simple, visual format. Case managers will use these to explain any activities they recommend for participants and what each activity involves. This tool will remain at case managers’ desks as a stationary aid for all client intake sessions. The tool responds to a need to improve client understanding of what participating in WF activities involves and to clarify the respective roles of DSHS and WF activity providers.
The second component is a client postcard designed to help participants link any activities to which they have been assigned to the goals elucidated during the CE (regarding employment, housing, family, etc.) and to make a plan for attending any assigned activities. The front of the postcard includes a goal-setting and reflection activity. This requires case managers and clients to facilitate a conversation about clients’ goals for themselves and their families and to articulate how participation in their WF requirements aligns with those personal goals.
The third component is the activity on the back of the postcard, which summarizes key action steps required for participants to follow through on their participation plan. Case managers will help clients formulate a plan to meet their initial WF requirements (e.g., decide and write down a time to make childcare arrangements and a transportation plan) and encourage clients to post this postcard in a visible place at home.

Data Collection

This ICR describes data collection related to phase 4 of the project described in the Generic Clearance.

We will collect information from staff (including supervisory staff) and clients for this IC. In the rest of this document and in Supporting Statement B, we include a description of:

Planned qualitative data collection (see Appendices A, B.1, and B.2 for the specific instruments). Instruments include focus group/interview protocols for DSHS frontline staff, DSHS supervisors and leadership, and TANF clients.
Planned qualitative analyses. Audio recordings and notes from interviews will be analyzed for patterns and themes.
Administrative data that DSHS is already collecting and that the project will utilize.

Universe of Data Collection Efforts

The data collection detailed below provide a holistic view of client engagement with the WF program and staff experiences with the intervention, for which information is not available through administrative data alone. If it is more convenient for respondents, these discussions may be held in group settings. Each focus group discussion will include staff at the same or similar levels. For example, one focus group discussion may be held with multiple frontline workers, such as those conducting the CE. A separate discussion may be held with supervisors of frontline staff. These data will contribute to the implementation study.

Client Interviews (Appendix A)

The research team will interview study participants to determine whether the barriers identified during the diagnosis phase (phase 3) were addressed by the intervention materials, and to understand overall client experiences. In Washington State, we propose to interview up to 30 clients who participated in or applied to the TANF WF program. These interviews will help the research team to understand how well the intervention was implemented and to contrast the experiences of clients in the program and control groups. Speaking to clients provides insight into the drivers of participation in their assigned WF activities, the intervention’s primary outcome. Not all questions will be asked of each respondent, based on the participant’s background or experience. We will reduce burden by asking only relevant questions.

Frontline Staff Case Study Reviews and Interviews/Focus Groups (Appendices B.1)

We propose to interview up to 40 frontline staff (both program and control group-serving) across the 5 participating community service offices (CSOs). The research team will use discussions with frontline staff to shed light on which aspects of the intervention worked well and which did not, how the interventions changed client-staff interactions and client participation, if and how the intervention materials were used as intended by program group-serving staff, and the extent to which the intervention conditions varied from business-as-usual. Frontline staff discussions have two components:

Case Study Reviews (Appendix B.1.1)

The case study reviews will be conducted in individual discussions with up to 40 frontline staff (both program and control group-serving) across the 5 participating CSOs and ask staff to use particular case files to recall the Comprehensive Evaluation (CE) with that participant. Research staff will request staff to walk through pertinent case details (e.g., ages of children, prior work history, and goals articulated during the CE), participants’ assigned activities, and staff perceptions of participant responsiveness to the program. These case study reviews will help research staff understand how the intervention tools were implemented in particular cases (implementation fidelity) and to compare reports of core details elucidated in CEs delivered by program group-serving and control group-serving staff (treatment differential).

Semi-Structured Interviews or Focus Groups with Frontline Staff (Appendices B.1.2 and B.1.3)

In a separate discussion, research staff will conduct semi-structured interviews or focus groups with up to 40 frontline staff (both program and control group-serving) across the 5 participating CSOs. These semi-structured interviews or focus groups will allows us to understand staff practices more broadly, beyond their memory and impressions of the specific cases profiled during the case file review, in order to answer questions of implementation fidelity and treatment contrast.

TANF WF Supervisors and Leadership Interviews/Focus Groups (Appendix B.2)

In addition, semi-structured interviews or focus groups with up to 15 TANF WF supervisors, CSO leadership, and DSHS leadership. These staff, who will not be involved in case study reviews or discussions with frontline staff, will allow us to understand the broader DSHS TANF context (e.g., policy changes, staffing constraints), how this affected implementation, and whether the intervention materials led to any unexpected consequences. Discussions with WF supervisors and CSO leadership (up to 10 across the 5 participating CSOs) will be held separately from DSHS leadership (up to 5).

Existing Administrative Data

In addition to collecting data from staff and clients with interviews, we also intend to supplement this information with administrative data DSHS is already collecting. In addition, we intend to review a sample of copies of completed intervention materials and client participation plans. We do not anticipate either of these data collection activities to create additional burden, as we intend to ask the site to collect data as it currently exists.

Research Questions and Study Design

Table 1 presents research questions that will be addressed by information collected in this phase.

Impact Research.

Research Question.

What is the effect of using a bundle of behaviorally informed materials in the initial step of WF participation – the intake session that sets a plan for program participation – on immediate (30-day) and on longer-term (7-12 months) engagement in the WF program?

Study Design.

We will conduct a randomized controlled trial with TANF applicants determined eligible for the WF program. All clients attend the initial orientation and CE as part of their work requirements under the TANF program. There are five participating DSHS CSOs. Each will leverage the current Washington State DSHS workflow management software and Management Information System (MIS) that routes participants to available case managers to facilitate the assignment process. This system will route participants to either available staff who have the intervention materials and are trained to use them (program group), or available staff who do not have the materials and are not trained to use them (control group). We will get data on baseline characteristics at the point when TANF clients are assigned to a WF Specialist. These data will allow us to compare the two groups (program group and control group), and we will conduct an analysis to determine whether the DSHS workflow management system has done a good job of randomly assigning clients into the two groups. If one member of a family is assigned to the treatment or control groups, all members of that family will be assigned to the same group if they are later invited to a CE.

Implementation Research

Research Questions

To what extent were the interventions implemented with fidelity?
In what ways did the treatment conditions differ from the counterfactual, as implemented?
What challenges and barriers did the site experience with implementation?
What is the organizational structure and culture, and how does it support or hinder responses to the behavioral intervention?
What are participant perspectives on DSHS, staff, and the intervention? In what ways did the materials affect motivation or engagement?
What are staff perspectives on their work for DSHS, clients, and client response to the intervention? What aspects of the tools seemed particularly effective in increasing motivation and participation?

Study Design

We will conduct an implementation study to describe and document the intervention, how it operated, and provide information about the contrast in treatment between the research groups—both whether the planned contrast between the treatment and the control condition occurred (implementation fidelity) as well as how the treatment implemented actually differed from the status quo (treatment contrast). This information will be important for interpreting the findings of the impact study and to understand implementation of behavioral interventions aimed at system and organizational-level change.

We will gather information to answer these questions through interviews with clients and either interviews or focus groups with staff. The qualitative data collection activities are essential to conducting implementation research. Administrative data do not record staff or client perspectives about what motivates or hinders participation in TANF WF services. We anticipate using the frontline staff and client discussions to elicit specific examples of how the intervention materials were used and what aspects of using them contributed to behavior change. Control-serving staff and control client discussions will provide information to help us assess the treatment contrast. Lessons learned in the implementation research will then be disseminated to program staff at offices for the remainder of the evaluation. Please see Appendices A and B.1 for client and frontline staff discussion protocols.

Table 1: Research Question and Instrument Matrix

Research Questions	Administrative data/MIS	Client interviews groups	Staff interviews/ focus groups Study Reviews
What is the effect of the intervention on case outcomes?	X
To what extent were the interventions implemented with fidelity?	X	X	X
In what ways did the treatment conditions differ from the counterfactual, as implemented?	X	X	X
What challenges and barriers did the site experience with implementation?	X		X
What is the organizational structure and culture, and how does it support or hinder responses to the behavioral intervention?			X
What are participant perspectives on DSHS, staff, and the intervention? In what ways did the materials affect motivation or engagement?		X
What are staff perspectives on their work for DSHS, clients, and client response to the intervention? What aspects of the tools seemed particularly effective in increasing motivation and participation?			X

A3. Improved Information Technology to Reduce Burden

We will use conference calls to the extent possible to minimize burden on staff. If a phone or video interview is easier for case managers' schedules, we will conduct interviews in that mode. Interviews will be recorded to reduce potential burden of participants restating responses.

Client and staff interviews will be scheduled at convenient times or when they are in the Washington State Department of Social and Health Services office for activities.

A4. Efforts to Identify Duplication

We have worked carefully with DSHS to understand the data it routinely collects on clients in its caseload. None of the data currently collected by DSHS would allow us to assess whether or not our intervention (supplemental print materials) improves client understanding of the case process, participant perspectives, and/or their willingness to engage in the process. In addition, the study team will not collect information from the TANF agencies and staff that is available from existing sources.

A5. Involvement of Small Organizations

We do not anticipate any small organizations to be affected by these information collections. Nonetheless, we will schedule interviews at times that are convenient to participants in order to minimize disruption of daily activities.

A6. Consequences of Less Frequent Data Collection

Rigorous evaluation of innovative initiatives is crucial to building evidence of what works and how best to allocate scarce government resources. Not collecting information about the implementation and effect of the intervention would hinder the government’s ability to learn how interventions were implemented and why and to what degree the interventions achieved the outcome desired.

A7. Special Circumstances

There are no special circumstances for this data collection.

A8. Federal Register Notice and Consultation

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on May 23, 2017, Volume 82, Number 98, page 23572, and provided a 60-day period for public comment. No substantive comments were received during the notice and comment period.

The research team consulted with behavioral science experts on the intervention design and on select items in the data collection protocols.

A9. Incentives for Respondents

Clients participating in interviews will receive a gift card of $25, which is standard for clients in Washington who have participated in other data collection activities with the state. Gift cards will help offset the financial burden that may result from extenuating travel, additional cell-phone data or phone minutes, or child care costs associated with interviews. Clients involved with the Washington State Department of Social and Health Services are likely to be under some stress and potentially difficult to reach, especially given that they may have an unfavorable view of DSHS. The $25 gift card will help to offset these incidental costs associated with participation and the incentive is not so high as to be coercive for clients.

Incentives will not be used as a substitute for other best-practice persuasion strategies designed to increase participation, such as explanatory advance letters, endorsements by people or organizations important to the population being surveyed, and assurances of privacy.

We have secured MDRC’s Institutional Review Board’s (IRB) approval for the study and the use of incentives for interviews with clients.

The Washington State IRB deemed the scope of the impact study exempt from IRB approval, and both boards will review a modification with the implementation research detailed above, including the use of incentives for interviews with clients.

A10. Privacy of Respondents

All respondents who participate in research under this clearance will be read a statement that will explain the study and will inform individuals that their participation is voluntary and of the extent of their privacy as respondents (informed consents are included in each instrument in Appendices A and B). Participants will be told verbally that their conversations will not be shared in a form that identifies them to anyone outside the research team. As ACF’s prime contractor, MDRC plans to implement all data collection activities. Information will be kept private to the extent permitted by law and in accordance with current federal information security standards and other applicable regulations.

MDRC employees are required to maintain and process quantitative and qualitative data in designated project folders on the MDRC network. With the exception of the temporary storage of data during onsite collection, MDRC employees are not allowed to download, keep, or process individual-level data on the hard drives of their MDRC work stations or any other storage. Information will not be maintained in a paper or electronic system from which they are actually or directly retrieved by an individual’s personal identifier.

The project Data Manager will organize BIAS-NG project folders and will supervise storage of BIAS-NG data files on a “need-to-know” basis. Following standard MDRC practice, the project Data Manager and project programmers will replace all personally identifiable information (PII) from incoming source data with a randomly generated project ID number. Also these files will be saved in secure folders with limited access on a “need-to-know” basis. Thereafter, analysis will be performed on data files stripped of PII (we will initially retain PII in order to merge individual data sources to create an analysis file). All reports, tables, and printed materials are limited to presentation of aggregate numbers.

MDRC’s IRB has approved this impact study. The Washington State IRB deemed the scope of the impact study exempt from IRB approval, and both boards will review a modification with the implementation research detailed above.

A11. Sensitive Questions

We are asking some sensitive questions in this data collection in terms of client relationships with staff and vice versa. For example, we ask clients in an interview if there is anything their case manager could have done differently to engage or interest them in WF. These answers will help the study team identify barriers to and facilitators of engagement. We assure clients that their case managers will not see their responses, and vice versa, to encourage honest responses.

A12. Estimation of Information Collection Burden

Table 2 provides details about how this estimate of burden hours and costs were calculated.

We expect to speak with up to 30 clients, 40 frontline staff, and 15 staff in leadership positions across the 5 participating CSOs. We expect to conduct both case study reviews and interviews/focus groups with up to 40 of the same frontline staff. We anticipate that case study reviews will take at most 1 hours to complete and frontline staff interviews/focus groups will take at most 1.5 hours to complete, for those same staff. These activities are collapsed into one response of 2.5 burden hours per frontline staff member in the table below. Client interviews and leadership interviews/focus groups will take at most 1 hour to complete.

The estimate below represents an upper bound on potential burden. We calculated the overall burden per respondent by multiplying the frequency of response by the time to complete each data collection item among respondents.

Table 2: Burden Hours for Washington State

Instrument	Total Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Total Burden Hours	Average Hourly Wage	Total Cost
Client interviews (Appendix A)	30	1	1	30	$12.00	$360.00
Case Study Review *and* Frontline Staff Interviews/Focus Groups (Appendix B.1)	40	1	2.5	100	$23.10	$2,310.00
Leadership Staff Interview/Focus Groups Only (Appendix B.2)	15	1	1	15	$23.10	$346.50
Totals	85			145		$3,016.500

Total Cost

To compute the total estimated cost for staff, the total burden hours were multiplied by the average hourly wage of “Community and Social Service Occupations” taken from the U.S. Bureau of Labor Statistics, May 2017 National Occupational Employment and Wage Estimates ($23.10).² To compute the total estimated annual cost for clients, the total burden hours were multiplied by $12.00, the 2019 Washington State minimum wage.³ The estimated total cost is $3,016.50.

A13. Cost Burden to Respondents or Record Keepers

There are no additional costs to respondents.

A14. Estimate of Cost to the Federal Government

The total cost for the data collection, analysis and reporting activities under this current request will be approximately $500,000.

A15. Change in Burden.

This is an individual Generic IC under the BIAS-NG Generic Clearance (0970-0502).

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

Phase 4: Evaluation

Phase 4 consists of implementing the behavioral intervention(s) and evaluating them and collecting long-term outcomes. Data collection will take place at different times following OMB approval. Implementation data from administrative data and client surveys will begin 1 month following approval. Implementation data from interviews will take place 1 month following OMB approval. Analysis will begin 3 months after OMB approval.

Phase 5: Dissemination

Dissemination efforts during the time of this clearance include site specific reports, infographics, dissemination products aimed at practitioners, sharing findings at conferences, and publicizing our findings and our work on social media. Dissemination efforts are expected to begin after analysis concludes (about 8 months after OMB approval).

A17. Reasons Not to Display OMB Expiration Date

All instruments will display the expiration date for OMB approval.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

1 The CE is part of the standard Washington State DSHS TANF applicant intake session.

2 U. S. Department of Labor, Bureau of Labor Statistics “May 2017 National Occupational Employment and Wage Estimates: United States.” Occupational Employment Statistics. 2017. https://www.bls.gov/oes/2017/may/oes_nat.htm#21-0000.

3 Washington State Department of Labor and Industries, “The Current 2019 Washington Minimum Wage is $12.00,” Workplace Rights, Minimum Wage. https://www.lni.wa.gov/WorkplaceRights/Wages/Minimum/

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	0000-00-00

Supporting Statement A - BIAS NG Washington State TANF Site

BIAS-NG Generic Supporting Statement A_ Washington_3-15-2019.docx

ACF Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) Project