BIAS-NG Generic Supporting Statement B_third domain_to submit_final

ACF Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) Project

Generic Data Collection (OMB Number 0970-0502)

SUPPORTING STATEMENT PART B

Revision June 2019

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

TABLE OF CONTENTS

B1. Respondent Universe and Sampling Methods 2

B2. Procedures for the Collection of Information 7

B3. Methods to Maximize Response Rates and Deal with Non-response 8

B4. Tests of Procedures or Methods to be Undertaken 9

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 9

The data collections under this umbrella generic clearance consist of a series of mixed-methods studies to identify, develop, and test interventions related to social service and benefit receipt in the program areas of Temporary Assistance for Needy Families (TANF) and Child Welfare. This request for a revision is to add the program area of Early Head Start/Head Start (EHS/HS).

Behavioral Interventions to Advance Self-Sufficiency-Next Generation (BIAS-NG) seeks to gather information from state and local agencies and their clients and staff to inform intervention design and evaluation. Each of the proposed studies under this Information Collection Request (ICR) will involve a focused scope and moderate-sized samples.

B1. Respondent Universe and Sampling Methods

Target sites for this series of studies consist of regional, state, and local agencies providing services under the auspices of three ACF programs: TANF, Child Welfare, and EHS/HS. Potential sites are identified through two primary avenues: 1) Referrals from ACF program office staff familiar with state and local human services programs; 2) Interest expressed voluntarily by staff from human services programs while attending presentations on the BIAS project at public meetings and conferences. After potential sites are identified, based on referral and/or interest, we review available information such as marketing and recruitment materials, program manuals, and organization charts. Based on available information, the research team and ACF selects organizations for fieldwork to ensure a mix of program areas, populations, locations, and service approaches. To date, the research team and ACF have selected 3 TANF sites and 2 child welfare sites, and anticipate selecting up to two EHS/HS organizations for participation in the study.

The target respondents included in these generic information collections (GenICs) include but are not limited to:

• Customers or individuals receiving services from regional, state, and local ACF programs in the domains of TANF, Child Welfare, and EHS/HS.

• Staff working in regional, state, and local programs or agencies in the domains of TANF, Child Welfare, and EHS/HS.

In studies covered under this generic ICR at Phases 1-3, obtaining probability-based samples to reach the desired subpopulations of interest (e.g., foster parents, TANF clients, or EHS/HS families) will be cost-prohibitive and not needed for achieving study goals. Purposeful, targeted sampling through specific programs and other non-probability sampling designs will be used to develop a pool of potential respondents, potentially drawing from regional, state, and/or county caseloads.

Approved information collections under this clearance are underway in Allegheny County, PA (child welfare); Monroe County, NY (TANF); and Washington State (TANF). In total, over the 3 years this clearance covers (covering Phases 3 and 4), we anticipate meeting with up to 224 administrators, 885 frontline staff, and 885 clients in either interviews or focus groups. Additionally, we plan to field surveys to up to 5,100 clients and 567 staff across the same sites over the course of the two phases. The review of marketing and recruitment materials, program manuals, and organization charts helps determine the administrative staff or clients to include in focus groups or interviews. The research team aims for the standard 80 percent response rate among survey respondents. There will be an estimated total of 7,506 respondents (including administrator, staff, and client interviews, focus groups, and surveys) over the 3 years that this clearance covers.

Because the Phase 1 through 3 samples are not randomly selected, they may be biased and not fully represent the entire study population. For the first five selected sites (3 in TANF, 2 in child welfare), we have used purposive sampling to select potential participants for interviews and focus groups, and will continue this approach with HS/EHS. At early stages, interviews and focus groups are not being and will not be used as data collection, but to inform intervention design.

Once sites have been identified and interventions have been designed, there will be subsequent data collection for both the implementation and the impact studies undertaken during the evaluation stage, Phase 4. Three sites (2 in TANF and 1 in child welfare) have reached Phase 4 and each one has received approval for a separate GenIC. For the implementation studies, this data collection involves more formal interviews and/or focus groups. The implementation studies can also include surveys, which could use either random or purposive sampling, depending on the availability of information about the sampling frame, time, and resources. The limitations associated with purposive or any sampling method have been and will continue to be described in the Phase 4 submission, and will be clearly stated in any publications produced for this project. For the impact study, data collection has relied on and will continue to rely on administrative and/or MIS data, as described below.

Universe of Data Collection Efforts

Data collection activities at Phases 3 and 4 include:

• Administrator interviews/focus groups: In order to diagnose the problem, the research team gathers data during Phase 3 to better understand the barriers administrators see to full program implementation. In Phase 4, the research team returns to interview program administrators to determine whether the intervention was administered with fidelity and to help to determine the intervention’s effect on program administrators.

• Staff interviews/focus groups: Collecting information from program staff during Phase 3 helps the researchers better understand how the program operates, from the staff perspective and what barriers staff see to the program operating at its potential. Staff interviews/focus groups may incorporate prototyping activities.¹ Returning to staff during Phase 4 sheds light on what aspects of the intervention worked well and which didn’t work well from a staff perspective.

• Client interviews/focus groups: Interviewing clients during Phase 3 helps researchers better understand the barriers clients face when trying to access and interact with the program. Client interviews/focus groups may incorporate prototyping activities. These insights helps inform the interventions targeted at clients. Talking with clients during Phase 4 helps researchers better understand from a client perspective what is, or is not working with the intervention.

• Client survey: Surveying clients can provide researchers with a more representative sample of client opinions as to how the program operates both before (Phase 3) and after the intervention (Phase 4).

• Staff survey: Surveying program staff can provide researchers with a more representative sample of staff opinions as to how the program operates both before (Phase 3) and after the intervention (Phase 4).

• Administrative data: The research team has worked and will continue to work with sites to access administrative data the agencies are already collecting in the format in which the site collects it. This administrative data allows us to track proximal outcomes such as attendance at required appointments, submission of completed paperwork, or referral to additional services. This does not involve any burden on staff or clients.

• Direct observation: The research team acts as a “fly on the wall,” observing staff and client interactions. This observation does not involve any burden on staff or clients.

• Reviewing Case Files: As part of interviewing staff, the research team has asked and may continue to ask staff to share de-identified case files of their clients to better understand the needs of the clients they serve, how they document interactions, and the type of follow-up they engage in. This type of data collection does not impose burden because the team does not ask for the information to be provided in a specific format other than the one in which it already exists.

• MIS data collected: If a site collects client data in an MIS system, the research team requests to obtain, with no burden to the staff, administrative data at the time of enrollment into the study (i.e., random assignment data) and tracking data in order to see implementation measures for our sample. This data is requested in the existing format used by the site.

• Collecting site documents: If site staff send written materials to clients, the research team has requested and may continue to request copies of these materials, at no additional burden to the staff.

Analysis Plan – Phase 4

Impact Analysis: The collection of the administrative data from each site allows us to conduct impact analysis of each intervention. We have used and will continue to use a factorial design for our impact evaluation. The sample sizes required for impact studies are based on the assumptions that most tests use either a standard two group design or a 2x2 factorial design and the main outcome will be binary (i.e., percentage). This design provides an optimal balance between the complexity of the hypotheses that can be tested and the interpretability of the results. The standard statistical tests in this design are for main effects and an interaction. Main effects test the impact of one variable averaging across the levels of the other. The power for these tests is determined by the overall sample size of the study. Since we do not yet know the full set of interventions that will be undertaken, we provide power calculations that show the minimum sample size required to detect statistically significant true impacts with 80 percent likelihood.² Detailed plans for each study site will be included in Phase 4 submissions.

Table 1: Sample Size Estimates for Tests of Main Effects in 2x2 Factorial Designs and Simple Two-Group Design RCT

Notes: Calculations assume a two-tailed test, significance level of 10 percent, 80 percent power and equal allocation of participants to levels. Estimates are based on a binary outcome (such as using a service); minimum detectable effects are percentage point increase from baseline. Continuous outcomes require additional assumptions.

Implementation Analysis: The collection of qualitative data from each site allows us to conduct implementation analysis of each intervention. The implementation study has described and will continue to describe and document each site’s intervention, how it operates, and provide information about the contrast in treatment between the research groups—both whether the planned contrast between the treatment and the control condition occurred (implementation fidelity) as well as how the treatment implemented actually differed from the status quo (treatment contrast). This information is important for interpreting the findings of the impact study. The implementation study will result in lessons for the field about how the interventions operate, the challenges they face, and the participants’ (clients and/or staff) perspectives on whether the behavioral interventions changed their behavior. Although specific components of the implementation study depend upon the sites’ specific behavioral interventions, our plan relies on a mixed-methods methodology, employing both qualitative and quantitative approaches.

Cost Analysis: The BIAS-NG project will include a cost analysis for all sites. While we could conduct a benefit-cost analysis, it would be important to have long-term follow-up for key outcome measures to monetize benefits. Thus, we will determine whether such an analysis is possible on a site-by-site basis. The cost analysis will estimate the per person cost of the intervention(s) over and above what is spent on the control group.

B2. Procedures for the Collection of Information

To gather information and inform intervention design, study teams composed of at least two members make the telephone and in-person meeting contacts. These staff members are experienced in the process of gathering information for purposes of designing demonstration and evaluation projects.

The study team sends each agency’s program director information about the study and offers the opportunity to speak with members of the study team. An overview email is addressed to program directors, when relevant, and introduces the study, its goals, and the team executing the proposed study on ACF’s behalf. Tailored emails may be included within individual generic ICRs. Attached to the email will be the project description. The study team is available to answer any questions about the study. When relevant for identifying a potential match between the study and a site, we may ask for select programmatic information, such as their administrative structure, experience, target population, and size, when relevant. We cover a set of topics relevant to the study and specific to the site to allow us to understand the variation of programming in the field, the range of perspectives on the BIAS-NG study, and whether particular study design options are feasible given the structure of the agencies’ programs. With a select group of programs, the study team follows up on any initial conversations with a request for further individualized discussion and data review to gain a better understanding of the program’s flow and solicit feedback about the potential interventions and study designs. Following initial analysis of these data to understand the flow of participants, the study team conducts an in-person visit to select sites. Subsequent visits and teleconferences are scheduled, as needed and with a narrower pool of programs, if the study team needs additional time to gather the information.

Once sites have been selected, we may conduct in-person focus groups and phone interviews to help ensure an effective design for the intervention. There are three separate protocols: one protocol to use for interviews and focus groups with staff who deliver services; one protocol to use for interviews and focus groups with administrators; and one protocol to use for interviews and focus groups with clients. The protocols in Appendix A provide an outline for the basic procedures that may be used for each data collection approach (i.e., focus group or individual interviews), the types of questions that may be asked and the expected flow of questions. Once finalized, instruments tailored to individual sites will be submitted through individual ICRs under this generic ICR.

Focus groups are facilitated by at least two individuals; one individual conducts the in-person interviews and surveys. Each focus group/interview begins with an introduction that explains the purpose and goals of the BIAS-NG project. Participants are then asked to read and sign the consent form. The facilitator informs participants that the conversation will be audio-recorded but that they will be able to have any comments they do not want repeated removed from the recording. Lastly, the Paperwork Reduction Act is explained and the OMB number for this collection and the expiration date is provided. In the focus groups, once all participants have signed consent forms, the facilitator asks each participant to introduce themselves and then begins the conversation. For individual in-person and phone interviews, the facilitator simply begins the conversation with the participant after receiving consent. At the end of the interview, participants have received a gift card worth up to $20. If approved through this revision, moving forward they will receive a gift card worth up to $40, though $20 will still be the default amount. Specific information about proposed incentives has been and will continue to be included in individual generic information collection requests under this clearance.

The focus groups and interviews in Phase 3 are designed to be formative and exploratory. Human services program staff and clients possess procedural and tacit knowledge that will be vital for identifying areas where behavioral insights may have a high impact. We plan to spend a maximum of 60 minutes with each staff person during each site visit. This data collection is used only for descriptive purposes, not as part of an impact evaluation.

For subsequent data collection for implementation research to better understand how well interventions have been implemented, at Phase 4, focus groups and interviews have been and will continue to be structured. Not all questions are asked of each respondent, based on the participant’s background or experience. We reduce burden by asking only relevant questions. For descriptive or implementation research purposes, the study may field self-administered questionnaires, either online or in person, to agency or program staff. In a mixed-mode approach, the survey firm first attempts to survey each respondent by telephone. Field interviewers then attempt to interview respondents who cannot be contacted by telephone in person. Study participants can refuse to complete the survey, or refuse to answer any of the questions on the survey, and will not be penalized in any way. Examples of items and instruments are provided in Appendix A; once they are developed and prior to use in the field, tailored, site-specific instruments have been and will continue to be submitted with additional information as generic IC requests for each of the approximately 9 tests across up to seven sites, along with information about the associated intervention.

B3. Methods to Maximize Response Rates and Deal with Non-response

At this formative stage to inform intervention design and site selection, for focus groups and interviews, we have taken and will continue to take several steps to help ensure a high rate of cooperation among respondents. First, ACF federal staff has pre-existing collaborative relationships with many program sites. We leverage these relationships to help secure buy-in from site staff to both participate in, and aid deployment of, data collection tools. The ultimate aim of the BIAS-NG project is to provide program sites with lessons to improve their on-the-ground operations. Explaining to sites the benefits they will receive from this project may help persuade sites that their effort is worthwhile.

For subsequent data collection from selected sites, the research team also has worked and will continue to work closely with administrators and staff to develop recruitment strategies for clients and program staff for focus groups and interviews, particularly to make sure we gather a group that reflects a mix of experiences. As is usually the case with focus groups, we recruit at least double the number of people for each focus group with the anticipation that half will not attend.

Staff working with the program and control groups may be asked to complete a survey several months after the launch of the intervention. Based on the response rates for the staff surveys in Allegheny County, we expect around 50 percent of staff to complete the survey if we are not able to offer incentives to staff, and significantly higher if we are able to offer such incentives. For surveys, we have used and will continue to use established methods, such as sending reminders, setting completion deadlines, attempting to reach participants by phone after several failed attempts to obtain a response, and working closely with staff to maintain an accurate list of contact information.

To further increase the likelihood of participation, we also offer incentives to clients and potentially staff (if approved) participating in focus groups, interviews, and/or surveys, as discussed in Supporting Statement Part A.

B4. Tests of Procedures or Methods to be Undertaken

No pre-testing has been completed at this time and there are no plans to pretest focus group or individual interview protocols. It is possible that formative focus groups and interviews may inform the development of focus group and interview protocols for subsequent data collection and inform the development of survey instruments. We may pre-test surveys, if it is necessary to develop novel items or instruments for the project, with 9 or fewer staff from a similar program that will not be in the study.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting

and/or Analyzing Data

Kim Clum

Senior Social Science Research Analyst

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

330 C St. SW, 4th Floor

Washington, DC 20201

(202) 205-4922

Victoria Berk Kabak

Social Science Research Analyst

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

330 C St. SW, 4th Floor

Washington, DC 20201

(202) 401-5871

1 Prototyping involves showing two versions of materials to people, observing how they interact with each, and asking them to explain their reactions to each. Questions start out general and become more specific if there are particular words or phrases within the materials that the designer wants to focus on. Respondents may be asked whether they understand the program’s rules and what they are being asked to do, what aspects they do not understand, and if they have suggestions for changes to the materials.

2 Since power calculations to determine the required sample size for a factorial design are conducted essentially the same way as in a standard two-group RCT (Somers et al, 2014), following Bloom (1995), we use , the formula to calculate the minimum detectable effect to determine the sample size for specified MDEs. We do this because we will not know the baseline outcomes (π) from which our study will be based until the Program Area Domains and problems are selected. Solving [1] for n, yields:

[2]

Where, = the proportion of the study population that would have a value of 1 for the binary outcome in the absence of the program

= the proportion of the study sample that is randomly assigned to the treatment group

= minimal detectable effect, which is smallest true impact that an experiment has a good chance of detecting

M = a multiplier equal to 2.49, representing the statistical significance level of 0.10 and power of 0.80.

6