BIAS-NG Generic Supporting Statement B_extension_5.23.22 clean

ACF Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) Project

Generic Data Collection (OMB Number 0970-0502)

SUPPORTING STATEMENT PART B

May 2022

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

TABLE OF CONTENTS

B1. Respondent Universe and Sampling Methods 2

B2. Procedures for the Collection of Information 7

B3. Methods to Maximize Response Rates and Deal with Non-response 8

B4. Tests of Procedures or Methods to be Undertaken 9

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 9

The data collections under this umbrella generic clearance consist of a series of mixed-methods studies to identify, develop, and test interventions related to social service and benefit receipt in the program areas of Temporary Assistance for Needy Families (TANF), Child Welfare, and Early Head Start/Head Start (EHS/HS). This is an extension request with no changes aside from adding up to one new site and extending the timeline through May 31, 2025 to allow up to five sites to complete data collection.

Behavioral Interventions to Advance Self-Sufficiency-Next Generation (BIAS-NG) seeks to gather information from state and local agencies and their clients and staff to inform intervention design and evaluation. Each of the proposed studies under the BIAS-NG Project Overarching Generic will involve a focused scope and moderate-sized samples.

B1. Respondent Universe and Sampling Methods

Target sites for this series of studies consist of regional, state, and local agencies providing services under the auspices of three ACF programs: TANF, Child Welfare, and EHS/HS. Potential sites are identified through two primary avenues: 1) Referrals from ACF program office staff familiar with state and local human services programs, and 2) Interest expressed voluntarily by staff from human services programs while attending presentations on the BIAS project at public meetings and conferences. After potential sites are identified, based on referral and/or interest, we review available information such as marketing and recruitment materials, program manuals, and organization charts. Based on available information, the research team and ACF selects organizations for fieldwork to ensure a mix of program areas, populations, locations, and service approaches. To date, the research team and ACF have selected three TANF sites, three child welfare sites, and two EHS/HS organizations for participation in the study. We are requesting permission to add a potential additional site from one of these three domains.

The target respondents included in these generic information collections (GenICs) include but are not limited to:

• Customers or individuals receiving services from regional, state, and local ACF programs in the domains of TANF, Child Welfare, and EHS/HS.

• Staff working in regional, state, and local programs or agencies in the domains of TANF, Child Welfare, and EHS/HS.

In studies covered under this overarching generic at Phases 1-3, obtaining probability-based samples to reach the desired subpopulations of interest (e.g., foster parents, TANF clients, or EHS/HS families) will be cost-prohibitive and not needed for achieving study goals. Purposeful, targeted sampling through specific programs and other non-probability sampling designs will be used to develop a pool of potential respondents, potentially drawing from regional, state, and/or county caseloads.

The following information collections have been completed under this overarching generic:

1. Allegheny County, PA (Child Welfare)

2. Monroe County, NY (TANF)

3. Washington State (TANF)

4. LA County, CA (TANF).

The following Phase 3 (diagnosis and design) information collections are ongoing under this overarching generic:

1. Matrix/Starfish (EHS/HS)

2. Hennepin County (Child Welfare).

For Phase 3, the team plans to submit one new information request for a potential new site.

For Phase 4 (implementation and impact studies), the team plans to submit up to four additional information collections (LA County child welfare, Hennepin County child welfare, and Matrix/Starfish EHS/HS, and one potential new site).

In total, over the three years this extension request covers (covering Phases 3 and 4), we anticipate meeting with up to 144 administrators, 1,200 frontline staff, and 1,200 clients in either interviews or focus groups. Additionally, we plan to field surveys to up to 12,400 clients and 1,600 staff across the same sites over the course of the two phases. The review of marketing and recruitment materials, program manuals, and organization charts helps determine the administrative staff or clients to include in focus groups or interviews. The research team aims for the standard 80 percent response rate among survey respondents.

Because the Phase 1 through 3 samples are not randomly selected, they may be biased and not fully represent the entire study population. We have used and will continue to use purposive sampling to select potential participants for interviews and focus groups. Three sites in Phase 3 decided to conduct diagnosis research studies for Phase 3, which were approved by the Office of Management and Budget (OMB) in individual GenIC requests.

Once sites have been identified and interventions have been designed, there has been and will continue to be subsequent data collection for both the implementation and the impact studies undertaken during the evaluation stage, Phase 4. Four sites (3 in TANF and 1 in Child Welfare) have completed Phase 4 and each one submitted and received approval for a separate GenIC. For the implementation studies, this data collection involves formal interviews and/or focus groups. The implementation studies can also include surveys, which could use either random or purposive sampling, depending on the availability of information about the sampling frame, time, and resources.

The limitations associated with purposive or any sampling method have been and will continue to be described in any GenIC submission, and will be clearly stated in any publications produced for this project. For the impact study, data collection has relied on and will continue to rely on administrative and/or MIS data, as described below.

Universe of Data Collection Efforts

Data collection activities at Phases 3 and 4 include:

• Administrator interviews/focus groups: In order to diagnose the problem, the research team gathers data during Phase 3 to better understand the barriers administrators see to full program implementation. In Phase 4, the research team returns to interview program administrators to determine whether the intervention was administered with fidelity and to help to determine the intervention’s effect on program administrators.

• Staff interviews/focus groups: Collecting information from program staff during Phase 3 helps the researchers better understand how the program operates from the staff perspective and what barriers staff see to the program operating at its potential. Staff interviews/focus groups may incorporate prototyping activities.¹ Returning to staff during Phase 4 sheds light on what aspects of the intervention worked well and which didn’t work well from a staff perspective.

• Client interviews/focus groups: Interviewing clients during Phase 3 helps researchers better understand the barriers clients face when trying to access and interact with the program. Client interviews/focus groups may incorporate prototyping activities. These insights helps inform the interventions targeted at clients. Talking with clients during Phase 4 helps researchers better understand from a client perspective what is, or is not working with the intervention.

• Client survey: Surveying clients can provide researchers with a more representative sample of client opinions as to how the program operates both before (Phase 3) and after the intervention (Phase 4).

• Staff survey: Surveying program staff can provide researchers with a more representative sample of staff opinions as to how the program operates both before (Phase 3) and after the intervention (Phase 4).

• Administrative data: The research team has worked and will continue to work with sites to access administrative data the agencies are already collecting in the format in which the site collects it. This administrative data allows us to track proximal outcomes such as attendance at required appointments, submission of completed paperwork, or referral to additional services. This does not involve any burden on staff or clients.

• Direct observation: The research team acts as a “fly on the wall,” observing staff and client interactions. This observation does not involve any burden on staff or clients.

• Reviewing Case Files: As part of interviewing staff, the research team has asked and may continue to ask staff to share de-identified case files of their clients to better understand the needs of the clients they serve, how they document interactions, and the type of follow-up they engage in. This type of data collection does not impose burden because the team does not ask for the information to be provided in a specific format other than the one in which it already exists.

• MIS data collected: If a site collects client data in an MIS system, the research team requests to obtain, with no burden to the staff, administrative data at the time of enrollment into the study (i.e., random assignment data) and tracking data in order to see implementation measures for our sample. This data is requested in the existing format used by the site.

• Collecting site documents: If site staff send written materials to clients, the research team has requested and may continue to request copies of these materials, at no additional burden to the staff.

Analysis Plan – Phase 4

Impact Analysis: The collection of the administrative data from each site allows us to conduct impact analysis of each intervention. We have used and will continue to use a factorial design for our impact evaluation. The sample sizes required for impact studies are based on the assumptions that most tests use either a standard two group design or a 2x2 factorial design and the main outcome will be binary (i.e., percentage). This design provides an optimal balance between the complexity of the hypotheses that can be tested and the interpretability of the results. The standard statistical tests in this design are for main effects and an interaction. Main effects test the impact of one variable averaging across the levels of the other. The power for these tests is determined by the overall sample size of the study. Since we do not yet know the full set of interventions that will be undertaken, we provide power calculations that show the minimum sample size required to detect statistically significant true impacts with 80 percent likelihood.² Detailed plans for each study site will be included in Phase 4 submissions.

Table 1: Sample Size Estimates for Tests of Main Effects in 2x2 Factorial Designs and Simple Two-Group Design RCT

Notes: Calculations assume a two-tailed test, significance level of 10 percent, 80 percent power and equal allocation of participants to levels. Estimates are based on a binary outcome (such as using a service); minimum detectable effects are percentage point increase from baseline. Continuous outcomes require additional assumptions.

Implementation Analysis: The collection of qualitative data from each site allows us to conduct implementation analysis of each intervention. The implementation study has described and will continue to describe and document each site’s intervention, how it operates, and provide information about the contrast in treatment between the research groups—both whether the planned contrast between the treatment and the control condition occurred (implementation fidelity) as well as how the treatment implemented actually differed from the status quo (treatment contrast). This information is important for interpreting the findings of the impact study. The implementation study will result in lessons for the field about how the interventions operate, the challenges they face, and the participants’ (clients and/or staff) perspectives on whether the behavioral interventions changed their behavior. Although specific components of the implementation study depend upon the sites’ specific behavioral interventions, our plan relies on a mixed-methods methodology, employing both qualitative and quantitative approaches.

Cost Analysis: The BIAS-NG project will include a cost analysis for all sites. While we could conduct a benefit-cost analysis, it would be important to have long-term follow-up for key outcome measures to monetize benefits. Thus, we will determine whether such an analysis is possible on a site-by-site basis. The cost analysis will estimate the per person cost of the intervention(s) over and above what is spent on the control group.

B2. Procedures for the Collection of Information

To gather information and inform intervention design, study teams composed of at least two members make telephone and in-person meeting contacts. These staff members are experienced in the process of gathering information for purposes of designing demonstration and evaluation projects.

The study team sends each agency’s program director information about the study and offers the opportunity to speak with members of the study team. An overview email is addressed to program directors, when relevant, and introduces the study, its goals, and the team executing the proposed study on ACF’s behalf. Tailored emails may be included within individual GenICs. Attached to the email will be the project description. The study team is available to answer any questions about the study. When relevant for identifying a potential match between the study and a site, we may ask for select programmatic information, such as their administrative structure, experience, target population, and size, when relevant. We cover a set of topics relevant to the study and specific to the site to allow us to understand the variation of programming in the field, the range of perspectives on the BIAS-NG study, and whether particular study design options are feasible given the structure of the agencies’ programs. With a select group of programs, the study team follows up on any initial conversations with a request for further individualized discussion and data review to gain a better understanding of the program’s flow and solicit feedback about the potential interventions and study designs. Following initial analysis of these data to understand the flow of participants, the study team conducts an in-person visit to select sites. Subsequent visits and teleconferences are scheduled, as needed and with a narrower pool of programs, if the study team needs additional time to gather the information.

Once sites have been selected, we may conduct focus groups and phone interviews to help ensure an effective design for the intervention. There are three separate protocols: one protocol to use for interviews and focus groups with staff who deliver services; one protocol to use for interviews and focus groups with administrators; and one protocol to use for interviews and focus groups with clients. The protocols in Appendix A provide an outline for the basic procedures that may be used for each data collection approach (i.e., focus group or individual interviews), the types of questions that may be asked and the expected flow of questions. Instruments tailored to individual sites will be submitted through individual GenICs under this overarching generic.

Focus groups are facilitated by at least two individuals; one individual conducts the in-person interviews and surveys. Each focus group/interview begins with an introduction that explains the purpose and goals of the BIAS-NG project. Participants are then asked to read and sign or (if over the phone) verbally agree to the consent form. The facilitator informs participants that the conversation will be audio-recorded but that they will be able to have any comments they do not want repeated removed from the recording. Lastly, the Paperwork Reduction Act is explained and the OMB number for this collection and the expiration date is provided. In the focus groups, once all participants have completed the informed consent process, the facilitator asks each participant to introduce themselves and then begins the conversation. For individual in-person and phone interviews, the facilitator simply begins the conversation with the participant after receiving consent.

At the end of the interview, participants have received a gift card worth up to $40. $20-$25 has been the default amount. Based upon our experience in the field to date under this package, we also propose continuing to offer a small honorarium of $25 to program staff participating in future data collections under this overarching generic, as previously approved by the OMB and IRB. This is in recognition of the time and professional expertise they contribute to the studies. These honoraria are intended to both encourage staff participation and recognize their efforts to support a timely and high-quality data collection.

Specific information about proposed tokens of appreciation and honoraria has been and will continue to be included in individual GenIC requests under this clearance.

The focus groups and interviews in Phase 3 are designed to be formative and exploratory. Human services program staff and clients possess procedural and tacit knowledge that will be vital for identifying areas where behavioral insights may have a high impact. We plan to spend a maximum of 60 minutes with each staff person during each site visit. This data collection is used only for descriptive purposes, not as part of an impact evaluation. Thus far we have submitted and had approved diagnosis studies for three sites, which involve formal interviews and focus groups. One additional site may submit a GenIC for a diagnosis study. These methods allow the research team to ask questions about client and staff understanding of the current processes plus their perspectives on barriers and facilitators related to those processes, questions that cannot be answered through analysis of administrative data alone. The diagnosis research component can provide critical insights to designing an effective intervention, allowing the research team to properly diagnose ways in which agencies are not maximizing their impact for the populations they serve.

For subsequent data collection for implementation research to better understand how well interventions have been implemented, at Phase 4, focus groups and interviews have been and will continue to be conducted. Not all questions are asked of each respondent, based on the participant’s background or experience. We reduce burden by asking only relevant questions. For descriptive or implementation research purposes, the study may field self-administered questionnaires, either online or in person, to agency or program staff. Study participants can refuse to complete the survey or refuse to answer any of the questions on the survey, and will not be penalized in any way. Examples of items and instruments are provided in Appendix A; once they are developed and prior to use in the field, tailored, site-specific instruments have been and will continue to be submitted with additional information as GenIC requests for each of the approximately nine tests across up to nine sites, along with information about the associated intervention. Currently approved materials are available here: https://www.reginfo.gov/public/do/PRAICList?ref_nbr=201909-0970-003.

B3. Methods to Maximize Response Rates and Deal with Non-response

To inform formative stage intervention design and site selection, for focus groups and interviews, we have taken and will continue to take several steps to help ensure a high rate of cooperation among respondents. First, ACF federal staff has pre-existing collaborative relationships with many program sites. We leverage these relationships to help secure buy-in from site staff to both participate in, and aid deployment of, data collection tools. The ultimate aim of the BIAS-NG project is to provide program sites with lessons to improve their on-the-ground operations. Explaining to sites the benefits they will receive from this project may help persuade sites that their effort is worthwhile.

For subsequent data collection from selected sites, the research team also has worked and will continue to work closely with administrators and staff to develop recruitment strategies for clients and program staff for focus groups and interviews, particularly to make sure we gather a group that reflects a mix of experiences. As is usually the case with focus groups, we recruit at least double the number of people for each focus group with the anticipation that half will not attend.

Staff working with the program and control groups may be asked to complete a survey several months after the launch of the intervention. Based on the response rates for the staff surveys in Allegheny County, we expect around 50 percent of staff to complete the survey if we are not able to offer incentives to staff, and significantly higher if we are able to offer such incentives. For surveys, we have used and will continue to use established methods, such as sending reminders, setting completion deadlines, attempting to reach participants by phone after several failed attempts to obtain a response, and working closely with staff to maintain an accurate list of contact information.

To further increase the likelihood of participation, we also offer tokens of appreciations to clients and honoraria to staff participating in focus groups, interviews, and/or surveys, as discussed in Supporting Statement Part A.

B4. Tests of Procedures or Methods to be Undertaken

Since the start of the project, the data collection instruments have allowed the teams to answer the key research questions. Every GenIC includes materials that are based on experiences to date and if we identify necessary adjustments, we incorporate these as needed for each site (if adjustments are needed after OMB approval of a GenIC, those changes would be submitted for approval).

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting

and/or Analyzing Data

Kim Clum

Senior Social Science Research Analyst

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

330 C St. SW, 4th Floor

Washington, DC 20201

(202) 205-4922

Clint Key

Research Associate III

Center for Applied Behavioral Sciences at MDRC

200 Vesey Street

New York, NY 10281

(212) 340-4489

Lawrence Katz

Professor of Economics

Department of Economics, Harvard University

1805 Cambridge Street

Cambridge, MA 02138

(617) 495-7730

1 Prototyping involves showing two versions of materials to people, observing how they interact with each, and asking them to explain their reactions to each. Questions start out general and become more specific if there are particular words or phrases within the materials that the designer wants to focus on. Respondents may be asked whether they understand the program’s rules and what they are being asked to do, what aspects they do not understand, and if they have suggestions for changes to the materials.

2 Since power calculations to determine the required sample size for a factorial design are conducted essentially the same way as in a standard two-group RCT (Somers et al, 2014), following Bloom (1995), we use , the formula to calculate the minimum detectable effect to determine the sample size for specified MDEs. We do this because we will not know the baseline outcomes (π) from which our study will be based until the Program Area Domains and problems are selected. Solving [1] for n, yields:

[2]

Where, = the proportion of the study population that would have a value of 1 for the binary outcome in the absence of the program

= the proportion of the study sample that is randomly assigned to the treatment group

= minimal detectable effect, which is smallest true impact that an experiment has a good chance of detecting

M = a multiplier equal to 2.49, representing the statistical significance level of 0.10 and power of 0.80.

8