BIAS-NG Generic SSA_ Monroe County

Status of Study: This data collection is part of the Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) project. This new data collection pertains to Phase 4 (Evaluation) for the Monroe County site within the TANF domain. More information on the context of the BIAS-NG project can be found in Section A.1.1 – Study Background.
Evaluation and Measurement. This data collection is intended to inform the understanding of the implementation of an intervention being evaluated by a randomized control trial in this phase of the BIAS-NG project. This data collection includes interview and focus group protocols that will collect information about the treatment and control. The data collection also includes qualitative research protocols to collect practitioner feedback on these intervention materials.
Type of Study. The study features an implementation study (described here) and an impact study consisting of two randomized control trials (RCTs) to assess the impact of behaviorally-informed letters, magnets, text messages, and orientation on attendance of TANF.
Utility of the information collection. This research is motivated by the low attendance rates of TANF participants in required meetings and activities. This research aims to better understand the results from the randomized control trial. The results of the study and lessons learned from the design process can be shared with other practitioners in the TANF field and beyond to improve communication and engagement with social service clients. In addition, the RCT will be structured to build knowledge in the behavioral science field about the effectiveness of alternative communication approaches at spurring individual choices.

A1. Necessity for the Data Collection

The Office of Planning, Research, and Evaluation (OPRE) at the Administration for Children and Families (ACF), U.S. Department of Health and Human Services (HHS), seeks Office of Management and Budget (OMB) approval to conduct interviews and surveys with TANF program administrators, staff, and clients in Monroe County, NY to understand the mechanisms and effects of an intervention informed by behavioral science and intended to improve program outcomes. This information collection (IC) activity is planned as part of ACF’s Generic Clearance for the Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) project. The goal of the BIAS-NG Generic Clearance is to conduct qualitative and descriptive quantitative research to identify and understand the psychological and behavioral factors that can affect the effectiveness of human service programs in the areas of Child Welfare (CW) and TANF.

Study Background

The BIAS-NG project builds on a prior OPRE project, the Behavioral Interventions to Advance Self-Sufficiency (BIAS) project, which relied exclusively on administrative data to test the short-term impact of small “nudge” interventions in human services programs. Going beyond the work conducted for BIAS, the BIAS-NG project will test new interventions in more domains and collect a wider range of data.

The study described in this generic IC is launched in collaboration with the Monroe County Department of Social Services in Monroe County, NY. The behaviorally informed intervention is designed to improve engagement of TANF clients’ in certain mandated employment services. In addition to administrative data on outcomes, we plan to collect administrative data on implementation of the intervention as well as qualitative and quantitative information from program staff and participants to better understand the mechanisms and effects of behavioral interventions.

Legal or Administrative Requirements that Necessitate the Collection

There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose of Survey and Data Collection Procedures

Overview of Purpose and Approach

Context and Intervention

This Generic IC pertains to Monroe County, NY whose Department of Social Services handles TANF program services. This study is focused on the Office’s goal of improving client engagement in employment services under the TANF program.

Specifically, the intervention is designed to improve clients’ engagement in the Employment Assessment and Work Experience Program, both of which are mandated for clients who do not secure a job on their own within specified periods of time. These services are designed to identify employment skills, work opportunities, and provide placement in internships to build work experience Currently, the agency sends a letter, the contents of which are dictated by law, to clients notifying them of their obligation to attend the Employment Assessment and Work Experience Program. The letter explains when and where to appear for these meetings, but does not clearly explain each meeting’s purpose and length, or provide guidelines to clients on what they should do to plan for the meeting. Moreover, the letter is dense and may be confusing or intimidating to clients. We are testing an intervention that provides two stages of outreach and support to clients plus a revised presentation for those individuals in the Work Experience Program orientation, all designed to increase show up rates in required TANF activities.

First, program group clients who are invited to the Employment Assessment will receive a modified appointment letter to replace the original letter and a refrigerator magnet. The modified letter is personalized, and uses graphics and simple language to highlight the meeting date, time, location, and what to expect. Clients can check boxes on the revised form to prompt planning on how they will attend the meeting. Overall, the new letter provides visually engaging and clear information which the existing letter does not, and presents information in a simpler manner than the existing letter. To encourage recipients to put the letter on a refrigerator, a magnet, with a friendly message encouraging engagement and the county logo, will be mailed with the letter.

Additionally, clients with active cell phones will receive two text messages that reinforce the letter and remind clients to come to the Employment Assessment. These messages are sent 7 days and 2 days before the meeting. The text messages provide links to a map of the meeting location and a calendar appointment that can be added to a smart phone calendar. The first text reminder is designed to encourage planning to attend, the second text serves as a more proximate reminder to attend.

Program group clients who are invited to the Work Experience Program will also receive a second redesigned letter which incorporates the same behavioral strategies included in the Employment Assessment materials. Families with active cell phones will also receive two reminder text messages for this second meeting, similar to the ones for the Employment Assessment meeting.

Finally, a random selection of clients will receive a revised Work Experience Program orientation, designed to encourage peer connections, identify strategies to confront challenges in the workplace, and share testimonials that emphasize the importance of internships.

Data Collection

This IC describes data collection related to phase 4 of the project described in the Generic Clearance.

We will collect information from staff (including supervisory staff) and clients for this IC. In the rest of this document and in Supporting Statement B, we include a description of:

Planned qualitative data collection (see Appendices A, B, and C for the specific instruments). Instruments include interview protocols for agency staff and clients and a short survey for agency staff.
Planned qualitative analyses. Audio recordings and notes from interviews will be analyzed for patterns and themes.
Administrative data that the agencies are already collecting and that the project will utilize.

Universe of Data Collection Efforts

Staff interviews/focus groups (Appendix A): The research team will use these interviews to shed light on what aspects of the intervention worked well and which did not, and how the interventions changed client-staff interactions, if at all. If it is more convenient for respondents, these discussions may be held in group settings. Each group discussion will include staff at the same or similar levels. For example, one group discussion may be held with multiple front-line workers, such as those conducting the Employment Assessment or Work Experience Program meetings. A separate group discussion may be held with supervisors of front-line staff. These data will contribute to the implementation study.

Client interviews (Appendix B): The research team will interview program clients to determine whether the barriers identified during the exploration phase were addressed by the intervention materials, and to understand overall client experiences. In Monroe County, we propose to interview 12-30 clients who participated in the Employment Assessment and Work Experience Programs. These interviews will help the research team understand how well the intervention was implemented. Not all questions will be asked of each respondent, based on the participant’s background or experience. We will reduce burden by asking only relevant questions.

Staff/administrator survey (Appendix C): The research team will survey relevant staff and supervisors about their attitudes and perceptions of clients and their work, to help the team understand the context for the intervention. This survey will help us identify any cognitive biases staff may have about clients and their work. The research team will use this survey to incorporate more staff perspectives and better inform the study implementation and results.

In addition to collecting data from staff and clients with interviews, and surveys, we also intend to supplement this information with administrative data the agencies are already collecting. Collecting administrative data on client outcomes for the impact evaluation will not impose a burden on respondents or record keepers, as we intend to ask the site to provide data as it currently exists. We have requested that these data be provided in whatever native format they are typically kept by the participating agencies, with as few modifications as possible to minimize any potential burden (see Supporting Statement B). In addition, we will not ask more than nine individuals to provide the administrative data.

Research Questions and Study Design

Table 1 presents research questions that will be addressed by information collected in this phase.

Impact Research Question:

What is the effect of offering clients revised outreach, including a new appointment letter and text reminders, on attendance at the Employment Assessment and subsequent Work Experience Program meetings? What is the effect of offering clients a revised Work Experience Program orientation on attendance at the subsequent work internship?

Impact Study Design:

We will conduct a randomized field trial for TANF clients invited to the Employment Assessment. The Work Experience Program is a meeting subsequent to the Employment Assessment, and clients will stay in their treatment or control group for the tests for both meetings. All clients attend the initial Employment Assessment as part of their work requirements under the TANF program, so we will not have separate sites for randomization. We will use a standard 50/50 randomization. If one member of a family is assigned to the treatment or control groups, all members of that family will be assigned to the same group if they are later invited to an Employment Assessment. For the Work Experience Program presentation, clients are randomized based on which days of the week they attend the presentation.

Implementation Research Questions:

To what extent were the interventions implemented with fidelity?
What challenges and barriers did the site experience with implementation?
What is the organizational structure and culture, and how does it support or hinder responses to the behavioral intervention?
What are participant perspectives on the agency, staff, and the intervention?
What are staff perspectives on their work for the agency, clients and client response to the intervention? What cognitive biases may be in play?

Implementation Study Design:

We will conduct an implementation study to describe and document the intervention, how it operated, and provide information about the contrast in treatment between the research groups—both whether the planned contrast between the treatment and the control condition occurred (implementation fidelity) as well as how the treatment implemented actually differed from the status quo (treatment contrast). This information will be important for interpreting the findings of the impact study.

We will gather information to answer these questions through interviews with clients; either interviews or focus groups with staff; and surveys with staff. The qualitative data collection activities are essential to conducting implementation research. Please see Appendices A,B, and C for interview and survey questions.

Table 1: Research Question and Instrument Matrix

Research Questions	Administrative data/ MIS	Client interviews groups	Staff survey	Staff interviews/ focus groups
What is the effect of the intervention on case outcomes?	X
To what extent were the interventions implemented with fidelity?	X	X	X	X
What challenges and barriers did the site experience with implementation?	X		X	X
What is the organizational structure and culture, and how does it support or hinder responses to the behavioral intervention?			X	X
What are participant perspectives on the agency, staff, and the intervention?		X
What are staff perspectives on their work for the agency, clients and client response to the intervention? What cognitive biases may be in play?			X	X

A3. Improved Information Technology to Reduce Burden

We will use conference calls and emails to the extent possible to minimize burden on staff. If a phone or video interview is easier for caseworkers' schedules, we will conduct interviews in that mode.

Client and staff interviews, along with staff surveys, will be scheduled at convenient times or when they are in the Monroe County Department of Social Services office for activities.

A4. Efforts to Identify Duplication

We have worked carefully with the agency to understand the data the agency routinely collects on clients in its caseload. None of the data currently collected by the agency would allow us to assess whether or not our intervention (the revised letters and the planned text messages to clients) improves client understanding of the case process, participant perspectives and/or their willingness to engage in the process. In addition, the study team will not collect information from the TANF agencies and staff that is available from existing sources.

A5. Involvement of Small Organizations

We do not anticipate any small organizations to be affected by these information collections. Nonetheless, we will schedule interviews at times that are convenient to participants in order to minimize disruption of daily activities.

A6. Consequences of Less Frequent Data Collection

Rigorous evaluation of innovative initiatives is crucial to building evidence of what works and how best to allocate scarce government resources. Not collecting information about the implementation and effect of the intervention would hinder the government’s ability to learn how interventions were implemented and why and to what degree the interventions achieved the outcome desired.

A7. Special Circumstances

There are no special circumstances for this data collection.

A8. Federal Register Notice and Consultation

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on May 23, 2017, Volume 82, Number 98, page 23572, and provided a 60-day period for public comment. No substantive comments were received during the notice and comment period.

The research team consulted with behavioral science experts on the intervention design and on select items in the data collection protocols.

A9. Incentives for Respondents

Clients participating in interviews will receive a gift card of $25, which is standard for clients in Monroe County who have participated in other data collection activities with the County. Our hope is that a gift card will help offset the financial burden that may result from extenuating travel, additional cell-phone data or phone minutes, or child care costs associated with interviews. Clients involved with the Monroe County Department of Social Services are likely to be under some stress and potentially difficult to reach, especially given that they may have an unfavorable view of the Department of Social Services. We expect a $25 gift card will help to offset these incidental costs associated with participation. We do not believe the incentive is so high as to be coercive for clients.

Incentives will not be used as a substitute for other best-practice persuasion strategies designed to increase participation, such as explanatory advance letters, endorsements by people or organizations important to the population being surveyed, and assurances of privacy.

We have secured Institutional Review Boards (IRB) approval for the study and the use of incentives for interviews with clients.

A10. Privacy of Respondents

All respondents who participate in research under this clearance will be read a statement that will explain the study and will inform individuals that their participation is voluntary and of the extent of their privacy as respondents (informed consents are included in each instrument in Appendices A, B, and C). Participants will be told verbally that their conversations will not be shared in a form that identifies them with anyone outside the research team. As ACF’s prime contractor, MDRC plans to implement all data collection activities. Information will be kept private to the extent permitted by law and in accordance with current federal information security standards and other applicable regulations.

MDRC employees are required to maintain and process quantitative and qualitative data in designated project folders on the MDRC network. With the exception of the temporary storage of data during onsite collection, MDRC employees are not allowed to download, keep, or process individual-level data on the hard drives of their MDRC work stations or any other storage. Information will not be maintained in a paper or electronic system from which they are actually or directly retrieved by an individual’s personal identifier.

The project Data Manager will organize BIAS-NG project folders and will supervise storage of BIAS-NG data files on a “need-to-know” basis. Following standard MDRC practice, the project Data Manager and project programmers will replace all personally identifiable information (PII) from incoming source data with a randomly generated project ID number. Also these files will be saved in secure folders with limited access on a “need-to-know” basis. Thereafter, analysis will be performed on data files stripped of PII (we will initially retain PII in order to merge individual data sources to create an analysis file). All reports, tables, and printed materials are limited to presentation of aggregate numbers. MDRC will destroy all paper records and electronic records containing PII when no longer needed for research purposes in accordance with funder and contractual requirements, as well as MDRC retention policies.

MDRC’s IRB has approved this study.

A11. Sensitive Questions

We are asking some sensitive questions in this data collection in terms of client relationships with staff and vice versa. For example, we ask clients in a survey to rate the information and preparation they received from Monroe County or Rochester Works staff. And we ask staff in a survey to indicate their perceptions of obstacles facing their clients. These answers will help the study team identify barriers to and facilitators of engagement. We assure clients that their caseworkers will not see their responses, and vice versa, to encourage honest responses.

A12. Estimation of Information Collection Burden

Table 2 provides details about how this estimate of burden hours and costs were calculated.

We expect to speak with a total of at least 12 and up to 30 clients and at least 12 and up to 30 staff. We anticipate extending the staff survey to 50 staff and supervisors with an expected response rate of 80%, or 40 respondents. The estimate below represents an upper bound on potential burden.

We calculated the overall burden per respondent by multiplying the frequency of response by the time to complete each data collection item among respondents. We anticipate that focus groups or interviews for staff and clients will each take at most 1 hour to complete.

Table 2: Burden Hours for Monroe County

Instrument	Total Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Total Burden Hours	Average Hourly Wage	Total Cost
Client interviews (Appendix B)	30	1	1	30	$15.55	$466.50
Staff survey (Appendix C)	40	1	0.2	8	$33.90	$271.20
Staff interviews (Appendix A)	30	1	1	30	$33.90	$1,017.00
Total	110			70		$1,754.70

Total Cost

To compute the total estimated cost for staff, the total burden hours were multiplied by $33.90, the mean hourly earnings for management, professional and related workers in the civilian workforce as reported by BLS NCS (2010).¹ To compute the total estimated annual cost for clients, the total burden hours were multiplied by $15.55, the mean hourly earnings for high school graduates reported by BLS NCS (2010). The estimated total cost is $$1,754.70

A13. Cost Burden to Respondents or Record Keepers

There are no additional costs to respondents.

A14. Estimate of Cost to the Federal Government

The total cost for the data collection, analysis and reporting activities under this current request will be approximately $500,000.

A15. Change in Burden.

This is an individual Generic IC under the BIAS-NG Generic Clearance (0970-0502).

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

Phase 4: Evaluation: Phase 4 consists of implementing the behavioral intervention(s) and evaluating them and collecting long-term outcomes. Data collection will take place at different times following OMB approval. Implementation data from administrative data and client surveys will begin 1 month following approval. Implementation data from interviews will take place 1 month following OMB approval. Analysis will begin 3 months after OMB approval.

Phase 5: Dissemination: Dissemination efforts during the time of this clearance includes site specific reports, infographics, dissemination products aimed at practitioners, sharing findings at conferences, and publicizing our findings and our work on social media. Dissemination efforts are expected to begin after analysis concludes (about 8 months after OMB approval).

A17. Reasons Not to Display OMB Expiration Date

All instruments will display the expiration date for OMB approval.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

1 U. S. Department of Labor, Bureau of Labor Statistics “National Compensation Survey: Table 1: Summary Mean hourly earnings and weekly hours for selected workers and establishment characteristics.” 2010. http://www.bls.gov/ncs/ocs/sp/nctb1344.pdf.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Clum, Kimberly (ACF)
File Modified	0000-00-00
File Created	2021-01-15