BEES_Supporting Statement A_June 2020_CLEAN

Building Evidence on Employment Strategies

for Low-Income Families

OMB Information Collection Request

0970-0537

Supporting Statement

Part A

July 2020

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer:

Tiffany McCormack and Megan Reid

• This is a non-substantive change request as part of the Building Evidence on Employment Strategies for Low-Income Families (BEES) study. This ICR was originally approved November 9, 2019 (OMB# 0970-0537).

• We will use a two-phased approach for our information collection requests. The first phase, which was submitted included recruitment materials, instruments specific to programs already identified for recruitment, and other materials that will not change based on recruitment. The second phase includes materials that reflect recruitment of specific programs.

• BEES will conduct multiple studies of innovative interventions aimed at improving employment outcomes, advances in the labor market, and economic security for low-income individuals and families.

• BEES will include impact and implementation studies. Impact studies will include RCTs, when possible. Some programs in earlier stages of development will only involve implementation studies.

• BEES will address gaps in the evidence base about employment services for disadvantaged populations in a number of domains, including individuals struggling with disability or mental health issues, involved in the criminal justice system, or facing substance or opioid use disorders. Information gathered from BEES is broadly intended to add to the current scientific research in this service area. Policymakers will gain a better understanding of the current innovative efforts to provide employment services to low-income populations and the efficacy of these interventions. This could influence their decision-making about interventions and policies to support. Practitioners will find ideas for their own program improvement and better understand the landscape of employment services for these populations.

Overview

A1. Necessity for the Data Collection

The Office of Planning, Research, and Evaluation within the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks approval to recruit evaluation sites and collect data from sites and participants for the Building Evidence on Employment Strategies for Low-Income Families (BEES) study. The overall purpose of BEES is to conduct rigorous evaluations of innovative interventions aimed at improving employment outcomes, advances in the labor market, and economic security for low-income individuals and families. Populations of particular interest for BEES include individuals facing substance and opioid use disorders, disability or mental health challenges, and their families; criminal justice-involved populations; or others at increased risk of enrolling in TANF or SSI/SSDI. While the target population and domains may vary, all interventions will include an employment services component. BEES will include impact and/or implementation studies of up to 21 different programs.

Study Background

There is both a great deal known, and a great deal more to learn, about interventions designed to increase employment and economic security among low-income Americans. Rigorous studies have demonstrated that many types of interventions can improve labor market outcomes for disadvantaged groups. Job search and subsidized employment can increase employment and earnings in the short-term, while earnings supplements can increase both employment and income, at least while supplements remain in place. Career pathways models can produce lasting earnings gains for those who meet program entrance criteria. But there remain many open questions on how employment interventions work for specific populations struggling with employment.

BEES will build ACF’s program of research in employment training and support strategies, providing new perspectives on employment for low-income individuals in a few specific domains. Below is an overview of the scientific research available on employment programs for low-income individuals within these domains: substance and opioid use disorder, criminal justice, and mental health and disability. These domains represent populations at risk of enrollment in TANF or SSI/SSDI. The summaries below indicate where the gaps in research lie about employment services for these examples of populations of interest to ACF, and how the BEES research questions may address these gaps.

1. Substance and Opioid Use Disorder. Introducing vocational services into drug treatment programs has been a widespread policy recommendation for decades. However, most substance use disorder treatment and recovery programs have excluded employment-focused services since most insurers do not reimburse for non-medical services, leading to a lack of research in this area. In addition, studies of employment training or counseling in drug treatment have rarely had adequate comparison or control groups and have therefore not provided reliable results. For the most part, studies have compared voluntary participants in employment activities with people who did not volunteer, introducing selection biases into the comparisons. The range of other predictors of ultimate success in both achieving drug reduction and employment have not been fully accounted for—age, education, prior work history, arrest background, housing status, ancillary problems such as co-occurring mental illness, and readiness or motivation. Promising approaches integrate treatment, employment services, and intensive supports in various combinations, but the evidence base on these models is thin, suggesting that there is a great deal to learn on this subject in BEES.

2. Criminal Justice. Prisoners returning to their communities after incarceration often face obstacles to regular employment, including low levels of education and lack of recent work experience (Pager, 2003; Holzer, Raphael, and Stoll, 2004; Uggen, Wakefield, and Western, 2005). In addition, some employers avoid hiring job applicants with criminal records. Ex-prisoners have low earnings upon reentry; in the twelve months after release, only 55 percent reported any earnings at all, and only 20 percent earned more than $15,000 during the year (Looney and Turner, 2018). The model of transitional jobs for those recently released from prison has had mixed results, with a lack of sustained employment and only one program showing reductions in recidivism rates (Jacobs Valentine, 2012; Redcross et al., 2012; Butler et al., 2012). Sustaining employment for this population continues to be a topic of interest, and one that could be addressed by BEES.

3. Mental Health and Disability. RCTs have found that improved access to and quality of mental health care reduces disability and improves employment outcomes (Wang et al., 2007). Integrating employment interventions with mental health treatment yields additional benefit (Lerner et al., 2015). Among people with mental health conditions, supported employment programs, like the Individual Placement and Support model (IPS), have positive effects on mental health symptoms and impairment (Mueser, Drake, and Bond, 2016). Implementing IPS in settings outside of the community mental health system for people with diverse disabilities is proving to be an especially fruitful area of study. Given these bi-directional relationships, it is important to consider going beyond the simple dichotomy between “treatment first” or “job first”. Instead, as in the substance abuse domain, organized interventions to improve mental health care and promote competitive employment may have synergistic benefits. Overall, there is strong evidence for the efficacy of specific treatment approaches for depression, anxiety and other conditions that can act as barriers to steady employment. The challenge – and a promising area of inquiry for BEES – is to learn how best to promote steady participation in treatment, and how to integrate treatment with employment services.

Building the evidence base in these and other areas will further ACF’s broad goal of increasing the number of employment-focused interventions supported by rigorous research. The broad intent of BEES is to increase the number of evidence-supported comprehensive interventions and to support the field in moving toward rigorous evaluations. While BEES is prioritizing interventions that have been fully implemented previously, undergone formative testing, or are candidates for an impact evaluation, other programs or interventions are also being considered. The overarching goal is to identify interventions that can move toward more rigorous evaluation. In some cases, this may mean an evaluation focused solely on implementation research.

BEES will include studies of up to 21 different programs, each focusing on improving employment outcomes for low-income individuals. Program services to be evaluated could include employment and training services such as job search assistance, job readiness services, vocational education, and coaching; employment barrier removal services such as those targeting substance use disorder and mental health; and other services aimed at promoting and supporting employment and economic security, consistent with the purposes of Temporary Assistance for Needy Families (TANF) and HHS priorities. Randomized control trial designs will be prioritized where an impact study is appropriate and possible. Any impact study will be accompanied by a corresponding implementation study, which will describe the programs and contextualize the impact results. For programs of interest that are not appropriate for an impact study, BEES will pursue an implementation-only study.

The study team used the generic clearance, Formative Data Collections for ACF Research (OMB #0970-0356), to learn more about programs of interest to inform the design of this larger study. Based on this information, a number of sites have been identified as candidates for recruitment. See Section A2 for additional information.

Coordination with Current OPRE Project and Social Security Administration

BEES is actively coordinating with another project sponsored by OPRE, the Next Generation of Enhanced Employment Services (NextGen) Project. The NextGen Project may include approximately nine rigorous evaluations of employment-focused programs; these will not overlap with programs selected for the BEES project. BEES and the NextGen Project have a common goal to foster stronger understanding of the types of interventions that can improve labor market outcomes for low-income individuals; however, the projects also maintain separate domains of focus. BEES is especially interested in evaluating interventions for individuals struggling with opioid dependency, abuse of other substances, and/or mental health issues, while the NextGen Project is especially focused on evaluating interventions that are market-oriented and/or employer-driven. Additional domains of focus may emerge as both projects complete knowledge development and identify potential sites for participation. OPRE is intentionally and strategically coordinating these projects in order to prevent duplication of effort; fully capitalize on the opportunity the projects afford for large-scale, rigorous evaluation; and advance the knowledge base regarding effective employment strategies for low-income, vulnerable populations.

In addition, both projects are involved in a joint effort with the Social Security Administration (SSA). SSA will provide demonstration program funds to ACF to support the addition of a disability focus in both projects. SSA funds may be used for both evaluation-related expenses and program development in sites selected for evaluation under either project.

In order to ensure BEES and the NextGen Project produce comparable results, and to meet SSA’s priorities across both projects, the projects’ information collection requests to OMB will include several common instruments and/or questions within instruments. We provide a table in Section A.2 that outlines commonality between the BEES and NextGen instruments.

Legal or Administrative Requirements that Necessitate the Collection

Although this specific data collection is not mandated, the Consolidated Appropriations Act of 2017 requires ACF to conduct research to understand the impact of TANF and other programs on employment, self-sufficiency, and other outcomes. The BEES project meets this requirement.

A2. Purpose of Survey and Data Collection Procedures

Overview of Purpose and Approach

The purpose of BEES is to build the evidence base of innovative approaches designed to boost employment and earnings for low-income individuals receiving or at risk of receiving TANF. BEES will include two broad types of sites, each providing employment services for low-income individuals. Behavioral health sites will include those providing substance or opioid use disorder treatment or recovery services, as well as those addressing mental health issues or working with people with disabilities. Non-behavioral health sites will prioritize innovative approaches to improving employment outcomes for low-income individuals, such as whole family approach interventions.

BEES will include impact and implementation studies. While BEES is prioritizing sites ready for randomized controlled trials (RCTs), the research team has already identified some sites that are in earlier stages of development and not suitable for rigorous evaluation (discussed below). For these types of sites, the team will conduct implementation-only studies. For sites participating in an impact evaluation, administrative data will also be used to estimate program impacts on key outcomes, such as employment and earnings.

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.  

Study Design

The BEES design has three primary components:

1. Site Selection and Recruitment. Recruitment of up to 21 programs into the study, involving phone calls and visits to potential programs. This recruitment effort will elicit programs to participate in the impact study and corresponding implementation study, as well programs not suited for rigorous evaluation at present, but which could be candidates for implementation-only study.

2. Impact study. Programs enrolled in BEES will use rigorous evaluation designs, with an emphasis on randomized control trials (RCT) where possible, though other designs may be considered, as needed (see the Research Design and Statistical Power subsection of Supporting Statement B for additional details). Outcomes for program and control group members will be compared at follow-up using the instruments described in this package. The internal validity of our estimates of these program impacts will be particularly strong using an RCT design.

BEES will be testing a diverse set of interventions designed to improve employment outcomes for low-income individuals, with a focus on specific target populations that have characteristics that may affect their ability to work. The findings are not designed to be generalized to a broader population but may be used to suggest interventions that could be tested further (e.g., with multiple sites or other populations) to support broader generalization. 

As an exploratory study, one limitation is that tests that take place in only one location might not represent the effects in other locations or with other target populations. A second limitation is that some of the interventions may include multiple components and it will be difficult to determine which individual components are effective or whether combinations of components interact with one another to produce larger or smaller effects. These limitations will be clearly stated in written discussions of study findings.

3. Implementation study. All 21 programs will participate in an implementation study. For those participating in an impact study, the corresponding implementation study will document the theoretical underpinnings of the intervention studied in the impact analysis, explore the operational structures that support program implementation, document implementation practices and staff and participant experiences, assess whether staff practices reflect each intervention’s core principles and components, and document control group services. Each intervention’s theory of change (often called a logic model) will guide our choices of topics to pursue in understanding how a program operates; whether the program is being implemented with fidelity to the logic model; and whether, how, and why a program does or does not achieve its expected outcomes. Crucially, the analysis will also examine the program-control treatment contrast. Finally, the analysis will contribute to practitioner promising practices or lessons for future replication or adaptation.

As previously noted, other programs will only participate in an implementation study. Programs that are not suitable for an impact study at this time due to their maturity or the small size of their service population could still provide valuable information to inform the field through an implementation study. The study will focus on describing the program’s context within its community, its goals and structure, any partnerships it may have, and how it enrolls participants and delivers services. It will not look at participant outcomes as part of the study. This study component has the primary goal of adding to our knowledge about the landscape of employment services for low-income individuals, particularly in cases where rigorous evaluation is not currently feasible due to size or other concerns. Studies of implementation-only site will also examine whether there are options to scale up or enhance the services and potentially implement an impact study at a later date.

Phased Approach to Data Collection Approval

Formative Data Collections Already Completed

Using the generic clearance Formative Data Collections for ACF Research (OMB control number 0970-0356), the team has identified some programs that, with OMB approval, could be formally recruited for the BEES evaluation. The below table briefly summarizes programs that were identified under the formative data collection and have been formally recruited for the BEES project.

Type of Program (Number of Programs)	Intervention	Capacity for Evaluation	Proposed Study
Substance Use Disorder (SUD) Treatment and Employment Programs (6 programs)	Combines employment services with treatment and recovery services	This approach is relatively new to the field; most programs are small and still under development. Because of their size and maturity, these programs are not appropriate for a rigorous RCT, but may benefit from implementation studies.	Implementation-Only
Substance Use Disorder (SUD) Treatment and Employment Programs (2 programs)	Combines employment services with treatment and recovery services	These programs are larger and more mature and thus under consideration for a rigorous evaluation.	Impact and Implementation
Whole Family Approach Program (1 program)	Combines housing vouchers with employment services to improve two-generation outcomes, including employment and family well-being	This intervention provides the opportunity to build on a recently tested two-generation model that has positive early results.	Impact and Implementation
Non-Custodial Parent Program (1 program)	Combines employment services, parenting workshops and intensive case management for non-custodial parents interacting with child support agencies for the first time.	This intervention has been previously piloted and builds on evidence-based approaches for the model components.	Impact and Implementation
Health & Disability (4 programs)	Pairs employment services with existing health-related treatment services	These interventions will layer evidence-based approaches on other treatment services within well-established programs.	Impact and Implementation
Criminal Justice Involvement (2 programs)	Offer employment services in coordination with other existing services to people involved with the criminal justice system.	These interventions are currently being implemented or will draw on evidence-based approaches for implementation.	Impact and Implementation
Criminal Justice Involvement (1 program)	Addresses key barriers to employment associated with justice system involvement fines and fees	This issue is of great interest to the field, but the smaller program is not ready for a rigorous evaluation	Implementation-Only

Data Collections for Impact and Implementation Studies

As noted in the Overview, BEES is using a two-phased approach for OMB approval of the impact and implementation studies.

Previous Request: Phase I

In Phase I, we requested approval for materials to allow the research team to formally recruit the eleven programs identified through the approved generic clearance (discussed above) and continue to identify and recruit additional programs. Other instruments included under Phase I were instruments already tailored for the identified programs and instruments with items we do not anticipate changing based on further recruitment of programs.

Specifically, the request covered 1) site identification and recruitment instruments, 2) instruments we do not plan to revise based on future identified programs and 3) final instruments tailored for eleven identified programs. This included the following data collection activities (see the Study Design and the Universe of Data Collection Efforts subsections for additional details).

1. Site Selection and Recruitment

   1. Discussion Guide for National Policy Experts and Researchers (Attachment A)

   2. Discussion Guide for State and Local Administrators (Attachment B)

   3. Discussion Guide for Program Staff at Potential Sites (Attachment C)

2. Impact Studies

   1. Baseline Information Form for Participants (Attachment D)

   2. Contact Update Request Form (Attachment E)

3. Implementation Studies

   1. Program Managers, Staff, and Partners Interview Guide – SUD Programs (Attachment F)

   2. Program Managers, Staff, and Partners Interview Guide – Whole Family Approach Programs (Attachment G)

This request also included the following supplementary materials:

1. Informed Consent Form for Participants (Attachment H)

2. Welcome Letter (Attachment I)

Current Request: Phase II

Under Phase II, we have updated this initial Information Collection Request (ICR) to request approval for the remaining instruments. Under Phase I, we have identified and recruited remaining programs and are submitting, updated materials and burden estimates as a non-substantive change request.

The Phase II request includes the following data collection activities (see the Study Design and the Universe of Data Collection Efforts subsections for additional details).

1. Impact Studies

   1. 6-Month Follow-Up Participant Survey (Attachment J)

   2. 12- to 18-Month Follow-Up Participant Survey (Attachment K)

2. Implementation Studies

   1. Program Managers, Staff, and Partners Interview Guide (Attachment L)

   2. In-Depth Case Study of Staff-Participant Perspectives

      1. Participant Case Study Interview Guide (Attachment M)

      2. Program Staff Case Study Interview Guide (Attachment N)

   3. Program Staff Survey (Attachment O)

This request will also include the following supplementary materials:

1. 6- and 12- to 18- Month Survey Advance Letters (Attachment P)

2. 6- and 12- to 18- Month Survey Email Reminders (Attachment Q)

3. 6- and 12- to 18- Month Survey Flyer (Attachment R)

Data Collection Timeline

We expect data collection to take place over a three-year period, following OMB approval. Using the instruments approved under Phase I, we expected site recruitment to take place for 9 months, as sites will join BEES on a rolling basis. Due to COVID-19, this recruitment timeline has been extended. For the implementation-only studies identified to-date, information collection was delayed due to COVID-19 but began in June 2020. For sites identified and recruited using the instruments approved under Phase I that are ready for rigorous evaluation, we began baseline data collection in March2020.

For additional programs recruited under Phase I, we are now submitting the Phase II instruments for OMB approval. As sites begin enrolling participants in the evaluation, the participant surveys will occur 6 and 12- to 18- months after each participant’s enrollment. Pending OMB approval, we expect this data collection to take place over 27 months. Administrative records will supplement the baseline data and follow up interviews and will be collected for a period of approximately three years. Implementation study activities for sites where an impact study will occur – including site visits, staff survey, and case studies – will begin after sites begin enrollment in the study (approximately 6 months after OMB approval). We expect this will take place over 9 months. To keep to this timeline as much as possible during the COVID-19 pandemic, we are using virtual approaches such as video conferencing, more frequent phone calls and remote data collection.

Research Questions

Key research questions for BEES are outlined in the table below. First, we identify a set of specific research questions for the implementation-only studies of SUD treatment/employment programs and whole-family approach programs. Then, we identify broad research questions for all other sites. .

These questions will allow the researchers to both describe the landscape of employment interventions for populations of interest in BEES and measure the effectiveness of employment programs selected for rigorous evaluation.

Research Question	Impact Study	Implementation Study	Site Recruitment Phase
What are the context and goals for establishing programs?		X
How are programs structured and operated including organizational partnerships, target group and recruitment strategies, employment service provision, and other support services?		X
What are the lessons from implementing the programs, including successes and challenges?		X
Which programs and/or promising practices warrant further study to determine effectiveness?		X
What is the impact of the specific interventions, both on measures of success in the first year and on longer-term measures of employment, well-being1, and economic security measured through survey and administrative data sources?	X
What can be learned about the impact of these interventions for important subgroups within the target populations (such as race, education-level, or socioeconomic status)?	X
Based on prior and emerging research, and current practice, what types of programs and approaches are most promising for the target population of each intervention?	X	X	X
Which specific interventions in each domain are ready to move to the next level of evidence?		X	X
What lessons on program implementation and cost can shed light on the impact results and help facilitate the expansion or replication of successful interventions?		X
What lessons can be drawn across the study about the design and implementation of successful interventions?	X	X

Universe of Data Collection Efforts

As previously discussed, we have requested approval for these instruments in two phases. The initial ICR was for approval of Phase I data collection activities to allow for formal recruitment of programs for evaluation and to begin evaluation. Under Phase II, we are requesting approval for additional data collection instruments. Draft versions of these Phase II instruments were submitted in the initial ICR for informational purposes. We have since finalized these draft versions and are now submitting them for approval. Going forward, we anticipate tailoring some Phase I and Phase II instruments to specific programs. An overview of both Phase I and Phase II materials is provided below.

Site Selection and Recruitment

As noted above, site selection instruments were approved under the generic clearance Formative Data Collections for ACF Research (OMB control number 0970-0356) in order to gather information about potential sites (initially on January 19, 2018, with revised clearance approval on August 14, 2018). We used the discussion guides approved under Phase I to formally recruit programs and for further program identification efforts.

The BEES evaluation team will recruit up to 21 programs. This recruitment will be done using a comprehensive stakeholder outreach effort. Site recruitment staff, working in teams of two or three, will meet in-person and by phone to discuss the evaluation with select informants, state and local administrators, and program staff. These visits and telephone calls will be used to collect information to determine which promising interventions are a good fit for the study and likely to participate, if selected by OPRE.

Approved Phase I Instruments:

1. Discussion Guide for National Policy Experts and Researchers (Attachment A). We will consult with regional and national organizations that work closely with relevant state and local programs in the targeted domains in order to identify promising interventions and programs. This high-level outreach may also include academic or other experts from the fields of interest. Consultations will be completed primarily via telephone. We expect the call to take about one hour to complete.

2. Discussion Guide for State and Local Administrators (Attachment B). As promising interventions operated by local and state administrators are identified through discussions with national policy experts and researchers, as described above, further outreach with state and local administrators will help identify promising sites. The study team will introduce administrators to the study and the semi-structured discussion guide will be used to learn about potential programs and assess the feasibility of launching evaluation activities within each local context overseen by the administrators. The study team will also assess state and local administrators’ willingness to help recruit the program staff involved in operating services at potential sites. We expect these discussions to take about two hours to complete.

3. Discussion Guide for Program Staff at Potential Sites (Attachment C). Through the consultations listed above or the study team’s existing connections, a series of telephone conversations and 1-2 day in-person recruitment visits will take place with the program staff directly operating innovative program models. A discussion guide will be used to organize discussions. The semi-structured guide will be used to learn about program administrative structures, programmatic experiences, community service contexts, and the feasibility of potential evaluation designs. Meetings will be scheduled primarily with sites individually (although not always in person) to understand their program flow, respond to questions and concerns, and discuss research design options. We expect these discussions to take about 2.75 hours to complete.

Impact Study Instruments and Consent

The goal of each piece of information from the participant is to estimate the effects of the interventions or to understand the experiences of individuals who receive program services. In total, the research team expects to enroll 18,600 individuals across multiple programs. Individuals seeking services from one of the sites being studied will be asked to complete the following activities.

Approved Phase I Instruments:

1. Baseline Information Form for Participants (Attachment D). This will include 15 minutes of questions and be completed by individuals electronically or on paper. Baseline information collected will be used to describe the sample, form subgroups, and increase the statistical precision of the impact analysis. As noted, this form includes the breadth of questions to be asked in the form. Most questions in certain indicated sections of the instrument, such as those about health and substance use, will only be included for the programs serving the relevant population. Similarly, most questions about housing preferences will only be asked for programs using whole family approaches.

2. Contact Update Request Form (Attachment E). This letter requests updated contact information from the participant, to increase likelihood of successful interview completion. We expect the form to take about 6 minutes to complete.

3. Informed Consent Form for Participants (Attachment H). An informed consent form will describe the study and what it means to participate. If willing, participants will sign the form.

As noted above, the below instruments will be submitted under Phase II; we are not seeking approval for these instruments at this time. We will submit updated materials and burden estimates as either a non-substantive change request or a revision with abbreviated public comment time, dependent on the level of changes and decision of OIRA. Drafts of these instruments and burden estimates are included here for informational purposes.

Proposed Phase II Instruments:

4. 6-Month Follow-Up Participant Survey (Attachment J). This survey will be 15 minutes long and administered using mixed mode fielding for the non-behavioral health sites. The survey will provide quantitative data on service receipt. Similar information will also be collected for the control group to define the treatment contrast. For the program group, these data can help describe how participants view the intervention, how they engage with program staff, and the services and activities they use.

5. 12- to 18-Month Follow-Up Participant Survey (Attachment K). This survey will be 30 minutes, using mixed mode fielding, in behavioral health sites (those with a substance or opioid use, mental health, or disability focus). The survey will include questions in the following areas:

   1. Service receipt. As in the 6-month survey, the 12- to 18-month survey will provide quantitative data on service receipt. Similar information will also be collected for the control group to define the treatment contrast. For the program group, these data can help describe how participants view the intervention, how they engage with program staff, and the services and activities they use.

   2. Employment-related outcomes. Although administrative records will provide information on quarterly earnings, the survey would collect information on jobs not covered by the unemployment insurance system and characteristics of jobs (such as, full-time vs. part-time, hourly wages, and so forth).

   3. Public assistance. Individuals will be asked whether they or anyone in their household received benefits of the following types: TANF, SNAP, Supplemental Security Income or Social Security Disability Income, unemployment insurance, workers’ compensation, WIC, energy assistance, and housing vouchers. For each type of assistance, they will also be asked how many months they or anyone in their household received benefits since they were randomly assigned.

   4. Health. This includes the SF-12 health survey about overall health status that has been found to be a strong predictor of use of health care resources and disability benefits.

   5. Behavioral health. Regarding depression, the survey will include the Kessler 6-item questionnaire. Regarding substance use, the survey will include questions from the Addiction Severity Index – Drug/Alcohol Use section, which is a validated screener used to identify those who have substance use problems.

Implementation Instruments

These implementation instruments will be tailored for each program design, target group, and local context, and the instruments presented here represent the universe of questions that could be included. The Phase I instruments described below were tailored to programs already identified at the time of the initial ICR. The Phase II instruments described below include the universe of all questions that may be asked of sites recruited since the initial ICR and any future sites.

Approved Phase I Instruments:

1. Program Managers, Staff, and Partners Interview Guide – SUD Programs (Attachment F). Implementation studies will be conducted in SUD treatment or recovery programs that integrate employment services. If it is not possible to hold an in-person visit, interviews will be conducted virtually via phone calls or video conference. Specific topics include: the choice of target groups; participant outreach strategies; employment, training, and support service provided; SUD treatment and recovery services; development or refinement of existing employment-related activities and curricula to serve the target group; how and why partnerships were established; strategies for engaging employers with a SUD population; and promising practices and challenges.

2. Program Managers, Staff, and Partners Interview Guide – Whole Family Approach Programs (Attachment G). Implementation studies will be conducted in programs that integrate employment services with a whole family approach. If it is not possible to hold an in-person visit, interviews will be conducted virtually via phone calls or video conference. Specific topics include: the choice of target groups; participant outreach strategies; employment, training, and support service provided; other services offered; development or refinement of existing employment-related activities and curricula to serve the target group; how and why partnerships were established; and promising practices and challenges.

Proposed Phase II Instruments:

3. Program Managers, Staff, and Partners Interview Guide (Attachment L). Staff interviews will be conducted during two rounds of implementation study visits for most sites. If it is not possible to hold an in-person visit, interviews will be conducted virtually via phone calls or video conference. During each visit, 90-minute semi-structured interviews of program staff and partners will explore staff roles and responsibilities, the provision of program services, and implementation success and challenges. The number of interviewees will vary by site depending on the organization staffing structure, however, program managers, program staff (i.e. case managers), and key partners will be interviewed, for an average of 10 interviewees Topics for the interviews include: program model and structure, staffing, program implementation, program components strategies and staff experiences, participant knowledge, awareness, participation, and views of program, use of services and incentives, and counterfactual environment.

4. In-Depth Case Study of Staff-Participant Perspectives: Participant and Program Staff Interview Guides (Attachments M and N). In-depth case studies will be conducted at up to 14 impact and implementation evaluation sites examining selected participants and their corresponding case manager to understand how program staff addressed a specific case, how the participant viewed the specific services and assistance received, and the extent to which program services addressed participant needs and circumstances. For each site, one-on-one interviews will be conducted separately with six participants (each 90 minutes in length) and with their respective case managers (60 minutes in length). Staff interviews differ from those described above because they will focus on how the staff member handled a specific case in contrast to how the program works overall. The case studies will provide examples of how the program worked for specific cases, and will enhance the overall understanding of program operations, successes, and challenges.

5. Program Staff Survey (Attachment O). Online staff surveys will be fielded to 20 staff at each site and will cover background and demographics, staff responsibilities, types of services provided by the organization, barriers to employment, program participation, and organizational and program performance. Each survey will take 30 minutes to complete.

Administrative Records

In addition to the instruments outlined above, the study will use administrative records to estimate program impacts on key outcomes such as employment and earnings. These may include records on employment and earnings (National Database of New Hires), public assistance (TANF, Supplemental Nutrition Assistance Program (SNAP)), Medicaid claims, income (IRS), criminal justice records, child support awards and payments, disability benefits from the Social Security Administration, and National Student Clearinghouse data. Exact records to be collected will depend on the nature of the intervention. Respondents will be informed of any records that will be used as part of this study.

National Database of New Hires (NDNH) data will be the main source of information on employment and earnings. In addition, we will use two to three state data sources, depending on the specific program and population being served. These could include the following:

• Public benefits records will provide monthly Temporary Assistance to Needy Families (TANF) and Supplemental Nutrition Assistance Program (SNAP) benefit amounts, both prior to and after random assignment. These data might also provide information on Medicaid eligibility and some kinds of TANF-funded “non-assistance” such as wage subsidies paid to employers.

• Medicaid claims will provide information on Medicaid enrollment and Medicaid-reimbursed health care. This will be an important source of information on behavioral health treatment for interventions targeted at substance users and those with mental health problems.

• Other administrative data sources. Depending on the intervention, we might seek to acquire child support and criminal justice records. We might use child support records for interventions targeted at noncustodial parents or criminal justice records for sites that serve a high proportion of former prisoners. We may additionally use disability benefits from Social Security Administration and education records from the National School Clearinghouse. Finally, we will explore the possibility of matching our sample to IRS tax data to obtain a comprehensive set of information on earnings, income, and tax credits such as the Earned Income Tax Credit and the Child and Dependent Care Credit. MDRC is currently using IRS data in other projects.

Coordination with Current OPRE’s NextGen Project and SSA

As noted in Section A.1, BEES is actively coordinating with OPRE’s NextGen project. Both projects aim to evaluate employment-focused programs for the same broad populations: low-income and vulnerable populations with complex barriers to employment. Though both projects target similar populations, each have priority focus areas. In terms of interventions, NextGen will prioritize interventions that are market-oriented and/or employer-driven and as such BEES will not evaluate such interventions. In terms of populations of interest, BEES will prioritize evaluating interventions for individuals struggling with opioid or other substance use disorder; NextGen may also include individuals with substance use disorder in evaluations, as populations with complex barriers to employment may face multiple overlapping barriers. Further, both projects have partnered with the Social Security Administration to support the addition of a disability focus in both projects. The projects are coordinating closely in program identification, outreach, and recruitment to ensure efforts are not duplicated and programs are not burdened by being contacted and/or considered by both teams.

In order to ensure BEES and the NextGen project produce consistent results that can be compared, and to meet SSA’s priorities across both projects, the projects’ information collection requests to OMB will include several common instruments and/or questions within instruments. The below table summarizes common items across both project’s participant surveys and informed consent form.

BEES Instrument/Attachment	BEES items that will be the same in Next Gen materials
Baseline Information Form (Attachment D) Benefit Receipt	BEES items F.1 – F5
Baseline Information Form (Attachment D) Disability Status	BEES items I.1 – I.7
Baseline Information Form (Attachment D) Health	BEES items J.1 – J.9
Informed Consent Form (Attachment H)	Regarding other research uses: The Social Security Administration will do additional research on [how BEES/NEXTGEN program affects] your earnings and receipt of disability benefits. They will do this research through 2040. They will use information such as your name, gender, date of birth, and Social Security Number to try and locate you in their records. They will only use your information to do research. The information will not be used to make decisions about any benefits you receive from the Social Security Administration, now or in the future. The Social Security Administration will not contact you directly.
12- to 18- month Follow-Up Participant Survey (Attachment K*) Service Receipt and Participation	BEES items A1a-e, A1k-l; A19
12- to 18- month Follow-Up Participant Survey (Attachment K*) Program Satisfaction	BEES item B1, B3, B4
12- to 18- month Follow-Up Participant Survey (Attachment K) Physical and Mental Health	BEES items D8a-f
12- to 18- month Follow-Up Participant Survey (Attachment K) Substance Use	BEES section E
12- to 18- month Follow-Up Participant Survey (Attachment K) Household Information, Income, and Material Hardship	BEES items F1C-F1C.15, F7

* These items can also be found in the 6-month survey (Attachment J).

A3. Improved Information Technology to Reduce Burden

This study will use information technology to minimize respondent burden and to collect data efficiently.

The baseline information form will be administered electronically (using a computer, tablet, or smartphone) when possible. Completing the form electronically will move the participant quickly through the form. We will also create a web page that would allow respondents to update their contact information easily, efficiently, and at a time most convenient for them.

For the 6-month and 12- to 18-month interviews, respondents will call a phone center for computer-assisted telephone interviewing (CATI). CATI reduces respondent burden by using skip logic to quickly move to the next appropriate question depending upon a respondent’s previous answer.

The online staff survey will be hosted on the Internet via a live secure web-link. This approach is particularly well suited to the needs of these surveys in that respondents can easily stop and start if they are interrupted and review and/or modify responses in previous sections.

A4. Efforts to Identify Duplication

Data being collected for BEES is not available in any other form in a consistent manner across the evaluation’s sites.

Although many employment programs assess participants in some of the outcomes and baseline characteristics being collected in BEES, those assessments differ by local program, and local programs will not collect similar information on control group members. To the extent possible, state and federal administrative data will be used to assess outcomes such as employment and receipt of public assistance, so the interviews of study participants are focused on outcomes that are not available from those administrative records sources.

Information that is being collected for implementation research is not expected to be available in any other form. For example, program sites are expected to vary to the extent that they use a management information system, and those systems are likely to not to include a common set of information across sites. However, the team will verify with each site that information being requested is available only through the surveys and qualitative interviews that are proposed.

As mentioned above, BEES is actively coordinating with another project sponsored by OPRE, the Next Generation of Enhanced Employment Services (NextGen) Project (see the Study Background subsection for additional details).

A5. Involvement of Small Organizations

We anticipate that some of the sites may be small community organizations.

We will use technology to minimize the burden on program staff. Participants will be able to complete the baseline information form and informed consent form on a laptop or tablet where possible rather than relying solely on the use of paper forms, reducing the staff time required for this data collection.

A6. Consequences of Less Frequent Data Collection

The BEES data collection aims to collect information only as frequently as needed to achieve the aims of the study. Eliminating any proposed data collection items would compromise our ability to address key research questions.

1. Baseline Information Form for Participants (Attachment D). The baseline information form will be administered once. Without it, we would be unable to verify that random assignment has yielded program and control groups similar in their observable background characteristics and in their baseline measures of outcomes. The baseline information form is also essential for describing the characteristics of the study sample.

2. 6-Month Follow-Up Participant Survey (Attachment J). For participants in a program without a mental health, disability, or substance use focus, the six-month survey will be administered once. The follow-up survey is essential for allowing us to estimate the program impacts across domains of interest.

3. 12- to 18-Month Follow-Up Participant Survey (Attachment K). For participants in a program with a mental health, disability, or substance use focus, the 12- to 18-month follow-up survey will be administered once. The follow-up survey is essential for allowing us to estimate the program impacts across domains of interest.

4. Program Managers, Staff, and Partners Interview Guide (Attachment L). Semi-structured interviews will be conducted with program staff at two points in time, when possible. We propose to conduct two rounds of implementation study visits to each site to help understand how each program matures and changes over time, thus necessitating data collection at two points. If it is not possible to conduct the visits in-person, visits will be conducted virtually via phone calls or video conferencing. Due to the expected timing of site start up, we will have enough time in the evaluation study period to conduct two rounds of site visits in 14 sites. These interviews will be critical to understand the implementation of each program and its context.

5. Participant and Program Staff Case Study Interview Guides (Attachments M and N). The case study interviews completed at up to 14 sites will provide a clearer understanding of how individuals move through the programs. This will add depth to the information gleaned from the interviews described above. For each site, individual interviews will be conducted with at most six participants (each 90 minutes in length) and with their case managers (60 minutes in length).

6. Program Staff Survey (Attachment O). The staff surveys will be fielded once to twenty staff members at up to 14 participating sites. These surveys will provide further staff perspective on topics such as organizational performance and staff responsibilities. It will complement the interviews and case studies described above.

A7. Special Circumstances

There are no special circumstances for the proposed data collection efforts.

A8. Federal Register Notice and Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on January 31, 2019, Volume #84, Number 21, page 740, and provided a sixty-day period for public comment. A copy of this notice is attached as Attachment T. During the notice and comment period, no comments were received.

Consultation with Experts Outside of the Study

No outside experts were consulted in designing the data collection package.

A9. Tokens of Appreciation

The BEES study is designed to test the effectiveness of a variety of employment services using a random assignment study design, with longitudinal follow-up. The size of the sample is set at study recruitment, and a high response rate is necessary to maintain statistical power to detect meaningful effects when measuring participant outcomes. In addition, the integrity of the study’s estimates requires maintaining similar response rates for the randomly assigned treatment and control groups and across demographic groups of central interest to the research study. Maintaining high response rates will be especially difficult in BEES because the study may include target populations, such as, individuals facing substance and opioid use disorders, disability or mental health challenges, criminal justice-involved populations, or others at increased risk of enrolling in TANF or SSI/SSDI, all of which are particularly difficult to maintain contact with over time. Their circumstances often result in frequent moves, short stays in hospitals or treatment centers, short periods of time living with others, and in some instances, homelessness. Because of the complex design and challenging study population, it is important to build respondent buy-in early in the study and retain as much of the sample as possible over time.

We propose including tokens of appreciation during the outreach, locating, and data collection process.² These are intended to show study participants that the study team appreciates their ongoing participation in data collection activities and offset any incidental costs of participation. In accordance with OMB guidelines, the team took several factors into consideration when determining whether or not to use tokens.³ Specifically, the team took into account design-specific threats to data quality, efforts to reduce non-response bias, the complexity of the study design, and study sample retention over the 12- to 18-month follow-up period. As described in Supporting Statement B, section B3, these are one part of a broader study design intended to build participant buy-in and maintain participation over time.

The team’s logic and proposed amount for specific tokens is as follows:

Longitudinal Surveys

We propose offering tokens for study participants at each stage of data collection related to the survey protocol. Three factors informed the study’s choice of the amounts for survey respondents:

1. Respondent burden, both at the time of the interview and over the life of the study;

2. Costs associated with participating in the interview at that time; and

3. Other studies of comparable populations and burden

The study team proposes the following token amounts for the BEES data collection related to structured interviews, to be provided as a gift card upon completion:

• $5 for updating contact information at 3 months, 6 months, or 9 months post enrollment.

• $15 for completing a 15-minute survey at 6 months post enrollment.

• $25 for completing a 30-minute 12- to 18-month (post enrollment) survey.

In designing the token structure, the team considered that the BEES population is expected to be highly mobile, thus maintaining good contact data is critical to follow-up contact efforts. This is the logic for the contact update incentives. The survey incentive amount increases commensurate with increased length of the follow-up period and the survey length. The team proposes $15 for a shorter follow-up survey 6 months after enrollment. We propose a $25 incentive—a modest increase—for the 12- to 18-month follow-up survey to reflect the longer survey protocol administered at that time. The team theorizes that tokens will be a particularly powerful tool for maintaining a high response rate in the control group, given that these sample members will not be engaged in program services.

To contextualize the proposed amounts, our incentive strategy is similar to that approved for longitudinal survey studies in other federal information collections. These studies represent vulnerable, highly mobile, low-income populations—including ex-offenders and non-custodial parents such as the Enhanced Transitional Jobs Demonstration (OMB control number 0970-413), participants with disabilities such as the Benefit Offset National Demonstration (OMB control number 0960-0785), and homeless families such as the Family Options Study (OMB control number 2528-0259) —similar to the target population for BEES. See more information in the table below.

In-depth Interviews:

For the interviews with program participants for the case studies providing the staff/participant perspective, it is important to secure individuals with a range of background characteristics to capture a variety of possible experiences with program services. Although participation in the interviews is voluntary, there are potential burdens placed on study participants. Without offsetting the direct costs incurred by respondents for participating in the interviews, such as transportation and child care, the research team increases the risk that only those individuals able to overcome potential financial barriers will participate in the study.

We propose providing $50 for completing a semi-structured 90-minute interview during program participation, to be paid by gift card upon completion. The participants will receive a $50 gift card to account for expenses such as transportation, and/or childcare that may otherwise prevent their participation in the study. For context, this amount is consistent with the $50 incentive that OMB approved for the impact study of MIHOPE home visiting programs (OMB control number 0970 - 0402). It is also consistent with incentives that OMB approved for other focus groups and interviews with a similar burden estimate. For example, participants were offered $50 for 90-minute case study interviews as part of the Evaluation of SNAP Employment and Training Pilots (OMB control number 0584-0604).We believe $50 is a reasonable amount for the time and cost associated with participation in these data collection activities, but is not so high as to appear coercive for potential participants.

STUDY	TYPE	POPULATION	TIME FRAME	LENGTH	AMOUNT	RESPONSE RATE
Enhanced Transitional Jobs Demonstration (OMB control number 0970-413)	Follow-up interview	Noncustodial parents and former prisoners (mainly male respondents)	12 months following study enrollment	45 minutes	$40	67-82 percent, across sites
Family Options Study (OMB control number 2528-0259)	Contact update requests	Low-income homeless families	Quarterly	5 minutes	$15	Varied by wave and site 25-04%
	Short surveys		6 and 12 months following study enrollment	15 minutes	$15	73.2 percent at 6 months 71.5 percent at 12-months
	Follow-up survey		20 months following study enrollment	60 minutes	$50	81.4 percent
Benefit Offset National Demonstration (BOND) (OMB control number 0960-0785)	Follow-up interview	SSDI beneficiaries, difficult to locate	12 months following study enrollment	45 minutes	$25	84 percent
Mother and Infant Home Visiting Program Evaluation (MIHOPE) (OMB control number 0970 - 0402)	Semi-structured interview	Low income mothers	3-4 years following study enrollment	90 minutes	$50	N/A
	In-home assessment		5 years following study enrollment	2 hours		70 percent
Evaluation of SNAP Employment and Training Pilots (OMB control number 0584-0604)	In-depth “case study” Interviews	Low-income population	During program period	90 minutes	$50	N/A

A10. Privacy of Respondents

The study team is committed to protecting the privacy of participants and keeping private the data that are entrusted to us, to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. Study participants will be asked to provide a Social Security number (SSN) to facilitate collecting administrative data from state and federal agencies and to help the study team locate them in the future, but individuals do not have to provide an SSN to be in the study. The study’s consent form is included as Attachment H.

The MDRC IRB will review each evaluation planned as part of BEES individually. The study is currently in the site identification phase. As programs agree to participate and develop a study design alongside the research team, IRB applications will be presented for review and approval.

Due to the sensitive nature of this research (see A11 for more information), the evaluation will obtain a Certificate of Confidentiality. The study team will be applying for this Certificate and will provide it to OMB once it is received. The Certificate of Confidentiality helps to assure participants that their information will be kept private to the fullest extent permitted by law.

The following privacy and data security measures will be in place to protect respondents’ privacy, including any personally identifiable information collected about them:

1. All data, including paper files, portable media (e.g., voice/video recordings) and computerized files, are kept in secure areas. Paper files and portable media are stored in locked storage areas with limited access on a need-to-know basis.

2. Merged data sources have identification data stripped from the individual records or encoded to preclude overt identification of individuals.

3. All reports, tables, and printed materials are limited to the presentation of aggregate numbers.

4. Compilations of individualized data are not provided to participating sites.

5. Confidentiality agreements are executed with any participating research subcontractors and consultants who must obtain access to detailed data files. These agreements are corporate forms and will not be distributed to respondents.

6. As specified in the evaluator’s contract, the research team shall use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard.  The research team will ensure that this standard is incorporated into the Contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information.

7. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations.

8. All research staff will be trained on appropriate privacy and data security matters.

Information will not be maintained in a paper or electronic system from which they are actually or directly retrieved by an individual’s personal identifier.

A11. Sensitive Questions

Some questions in the baseline information form, 6-month, and 12- to 18-month interviews may be sensitive for study participants. Individuals are being asked about their use of alcohol and drugs as well as about their mental health, particularly depression symptoms. These questions are necessary because a goal of the study is to understand the effects of employment interventions for individuals with opioid dependence, other substance abuse, or mental health issues. As noted in section A4, this information will not be available from other data sources.

Across all data collection, respondents will be informed by research staff prior to the start of the interviews or surveys that their answers will be kept private to the extent permitted by law, that results will only be reported in the aggregate, and that their responses will not affect any services or benefits they or their family members receive.

At the point of enrollment in the study, the informed consent form (Attachment H) will provide an overview of data collection efforts to expect during the course of the study. Staff obtaining consent from participants will be trained to answer questions about what it means to participate in the study. Participants who have agreed to be in the study will be asked to provide personally identifiable information on the baseline information form (Attachment D) including, for example, date of birth, address, and social security number. The Baseline Information Form likewise asks about current mental health, substance use, and disability. At follow-up, interviews (Attachments F and G) may ask sensitive questions as well. Both ask, for example, about substance use disorder and mental health treatment. The collection of personal identifiers is necessary for participant locating for follow-up interviews and to allow us to access and match administrative records data.

A12. Estimation of Information Collection Burden

Table A.1 shows the annual burden and cost of the data collection instruments and activities described in this ICR. The assumed wage rate is based on the May 2017 employment and wages from Occupational Employment Statistics survey from the Bureau of Labor Statistics (http://www.bls.gov/oes/current/oes_stru.htm). The rate used for researchers and policy experts, $60.47, is equivalent to management, scientific, and technical consulting services under SOC code 19-3011. The rate used for State and local administrators, $49.12, is equivalent to the local government managers under SOC code 11-1021. The rate used for program staff $24.44 is equivalent to local government workers under SOC code 21-1023. The average hourly wage of participants is estimated from the average hourly earnings ($4.92) of study participants in the Building Strong Families Study (Wood et al., 2010). These average hourly earnings are lower than minimum wage because many study participants were not working. We expect that this will also be the case for BEES study participants.

Table A. 1: Total Burden Previously Approved Under Phase I

Phase I included the following instruments 1) site identification and recruitment instruments, 2) instruments we do not plan to revise based on future identified programs and 3) final instruments tailored for eleven identified programs. The time period for this approval spans from November 2019 through November 2022.

Instrument	Total Number of Respondents	Annual Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Annual Burden Hours	Average Hourly Wage	Total annual cost
Attachment A. Discussion guide for national policy experts and researchers	10	3	1	1	3	$60.47	$181.41
Attachment B. Discussion guide for state and local administrators	55	18	1	2	36	$49.12	$1768.32
Attachment C. Discussion guide for program staff at potential sites	72	24	1	2.75	66	$24.44	$1613.04
Attachment D. Baseline information form for participants	12400	4133	1	0.25	1034	$4.92	$5087.28
Attachment E. Contact Update Letter and Form	7520	2507	1	0.1	251	$4.92	$1234.92
Attachment F. Program managers, staff, and partner interview guide – SUD Programs	60	60	1	1.5	90	$24.44	$2199.60
Attachment G. Program managers, staff, and partner interview guide – Whole Family Approach Programs	20	10	2	1.5	15	$24.44	$366.60
Estimated Annual Burden Total					1,495 hours		$12,451.17

Table A.2: Burden Remaining from Previously Approved Information Collection

The below table reflects the amount of burden still remaining for collection. For this information collection request, the remaining burden has been annualized over the new request time period.

Instrument	Total Number of Respondents	Annual Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Annual Burden Hours	Average Hourly Wage	Total annual cost
Attachment A. Discussion guide for national policy experts and researchers	7	2	1	1	2	$60.47	$120.94
Attachment B. Discussion guide for state and local administrators	52	17	1	2	35	$49.12	$1719.20
Attachment C. Discussion guide for program staff at potential sites	60	20	1	2.75	55	$24.44	$1344.20
Attachment D. Baseline information form for participants	12299	4100	1	0.25	1025	$4.92	$5043.00
Attachment E. Contact Update Letter and Form	7520	2507	1	0.1	251	$4.92	$1234.92
Attachment G. Program managers, staff, and partner interview guide – Whole Family Approach Programs	20	10	2	1.5	15	$24.44	$366.60
Estimated Annual Burden Total					1,383 hours		$9,828.86

Table A.3: Newly Requested Information Collections

The table below reflects the burden for information collection proposed in Phase II of this ICR.

Instrument	Total Number of Respondents	Annual Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Annual Burden Hours	Average Hourly Wage	Total annual cost
Attachment J. 6-month Follow-Up Participant Interview	2400	800	1	0.25	200	$4.92	$984.00
Attachment K. 12-month Follow-Up Participant Interview	5120	1707	1	0.5	853.5	$4.92	$4199.22
Attachment L. Program Managers, Staff, and Partners Interview Guide	280	93	2	1.5	140	$24.44	$3421.60
Attachment N. Program Staff Case Study Interview Guide	84	28	1	1	28	$24.44	$684.32
Attachment M. Participant Case Study Interview Guide	84	28	1	1.5	42	$4.92	$206.64
Attachment O. Program Staff Survey	300	100	1	0.5	50	$24.44	$1222.00
Estimated Annual Burden Total					2808.5 hours		$23,168.95

Table A.4: Newly Requested Information Collections

The table below reflects the burden both previously approved (Phase I) and currently requested (Phase II) information collections.

Instrument	Total Number of Respondents	Annual Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Annual Burden Hours	Average Hourly Wage	Total annual cost
Burden for previously approved, ongoing data collection (Phase I)
Attachment A. Discussion guide for national policy experts and researchers	10	3	1	1	3	$60.47	$181.41
Attachment B. Discussion guide for state and local administrators	55	18	1	2	36	$49.12	$1768.32
Attachment C. Discussion guide for program staff at potential sites	72	24	1	2.75	66	$24.44	$1613.04
Attachment D. Baseline information form for participants	12400	4133	1	0.25	1034	$4.92	$5087.28
Attachment E. Contact Update Letter and Form	7520	2507	1	0.1	251	$4.92	$1234.92
Attachment F. Program managers, staff, and partner interview guide – SUD Programs	60	60	1	1.5	90	$24.44	$2199.60
Attachment G. Program managers, staff, and partner interview guide – Whole Family Approach Programs	20	10	2	1.5	15	$24.44	$366.60
Burden for newly requested information collections (Phase II)
Attachment J. 6-month Follow-Up Participant Interview	2400	800	1	0.25	200	$4.92	$984.00
Attachment K. 12-month Follow-Up Participant Interview	5120	1707	1	0.5	853.5	$4.92	$4199.22
Attachment L. Program Managers, Staff, and Partners Interview Guide	280	93	2	1.5	140	$24.44	$3421.60
Attachment N. Program Staff Case Study Interview Guide	84	28	1	1	28	$24.44	$684.32
Attachment M. Participant Case Study Interview Guide	84	28	1	1.5	42	$4.92	$206.64
Attachment O. Program Staff Survey	300	100	1	0.5	50	$24.44	$1222.00
Estimated Annual Burden Total					2808.5 hours		$23,168.95

A13. Cost Burden to Respondents or Record Keepers

There are no additional costs to respondents.

A14. Estimate of Cost to the Federal Government

The total cost for the data collection activities for phases one and two is estimated to be $12,577,000. Annual costs to the Federal government will be $4,192,333 for the proposed data collection. The table below breaks down these costs.

Phase I and II Total Costs
Cost Category	Estimated Costs
Site Recruitment & Evaluation Start Up Work	$1,517,000
Participant Interviews	$5,475,000
Implementation	$1,184,000
Publications/Dissemination	$4,401,000
Total	$12,577,000

A15. Change in Burden

This request is for additional information collection to include 6- and 12- to 18-month follow-up surveys and implementation study interview guides. These Phase II instruments are necessary in order to complete the rigorous evaluations started using instruments approved under Phase I of this OMB request.

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

Analysis Plan

Estimating program impacts.

With plans for randomized controlled trials (RCT) in many sites, the starting point for the intent-to-treat impact analysis is to compare outcomes for all program group members and control group members. The precision of the estimates will be enhanced by estimating multivariate regression models that control for factors at baseline that may also affect the outcome measures. Such impacts are often referred to as “regression-adjusted” impacts. To increase precision, impact estimates are regression adjusted, controlling for baseline characteristics. Impacts will also be estimated for key subgroups to investigate whether the interventions have larger effects for some groups of participants (Bloom and Michalopoulos, 2011). In the main subgroup analysis, subgroups will be chosen using baseline characteristics, based on each evaluation’s target population and any aspects of the theory of change that suggest impacts might be stronger for some groups. Supporting Statement B details the survey response bias analysis approach, including weighting, if necessary. Statistical analyses will be conducted in SAS.

Analyzing Implementation Study data.

Notes from qualitative data collection will be imported into Dedoose, MDRC’s mixed-methods analysis software. Notes will be coded using a pre-specified coding scheme that accounts for the priorities of research questions and what we hope to learn from the process study. Quantitative data (interviews and surveys, for example) will undergo descriptive statistics analysis in SAS. If warranted, quantitative data may also be imported into Dedoose for analysis.

Time Schedule and Publication

The table below summarizes the data collection timeline described in previous sections and the publication of implementation and impact findings through briefs, reports, and peer-reviewed journal articles. As noted earlier,

Activity	Length of Activity	Timeframe Post OMB Approval
Site Recruitment	9 months	Months 1 - 9
Participant Interviews	27 months	Months 7 - 33
Implementation	14 months	Months 2 - 15
Publications/Dissemination	12 months	Months 25 - 36

We expect data collection to take place over a three-year period, following OMB approval.

Upon OMB approval, site recruitment will begin and is expected to take place over a 9-month period. Sites will join BEES on a rolling basis. Recruitment will entail telephone, email, and in-person meetings with potential programs to introduce BEES and discuss the feasibility of, and program interest in, participation.

As sites begin enrolling participants in the evaluation, the interviews will occur 6 and 12 to 18 months following each participant enrollment. Pending OMB approval, we expect this data collection will continue for just over two years.

Implementation study activities for sites where an impact study will occur–including site visits, staff survey, and case studies–will begin after sites begin enrollment in the study. Pending OMB approval, we expect this data collection effort will continue for 9 months. For the implementation-only studies identified to date, information collection will begin soon after OMB approval.

A17. Reasons Not to Display OMB Expiration Date

All instruments will display the expiration date for OMB approval.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

References

Berlin, M., Mohadjer, L., Waksberg, J., Kolstad, A., Kirsch, I., Rock, D., & Yamamoto, K. 1992. “An experiment in monetary incentives.” Proceedings of the Survey Research Section of the American Statistical Association, 393-398.

Bloom, Howard S., and Charles Michalopoulos. 2011. “When is the Story in the Subgroups? Strategies for Interpreting and Reporting Intervention Effects on Subgroups” Prevention Science, 14, 2: 179- 188.

Butler, David, Julianna Alson, Dan Bloom, Victoria Deitch, Aaron Hill, JoAnn Hsueh, Erin Jacobs

Valentine, Sue Kim, Reanin McRoberts, and Cindy Redcross. 2012. What Strategies Work for the Hard-to-Employ? Final Results of the Hard-to-Employ Demonstration and Evaluation Project and Selected Sites from the Employment Retention and Advancement Project. New York, NY: MDRC.

Council of Professional Associations on Federal Statistics. 1993. “Providing Incentives to Survey

Respondents: Final report.” Retrieved January 17, 2019 from http://www.copafs.org/reports/providing_incentives_to_survey_respondents.aspx.

Dillman, D. A., Smyth, J. D., & Christian, L. M. 2009. Internet, mail, and mixed-mode surveys: The

tailored design method. (3rd ed.) New York, NY: John Wiley & Sons.

Gueron, Judith M., and Edward Pauly. 1991. From Welfare to Work. New York, NY: Russell Sage Foundation.

Hamilton, Gayle. 2002. Moving People from Welfare to Work: Lessons from the National Evaluation of

Welfare-to-Work Strategies. Washington, D.C.: U.S. Department of Health and Human Services

and U.S. Department of Education.

Hamilton, Gayle, and Charles Michalopoulos. 2016. Job Search or Basic Education Participation First:

Which Improves Welfare Recipients’ Earnings More in the Long Term? New York, NY: MDRC.

Holzer, Harry J., Stephen Raphael, and Michael A. Stoll. 2004. “Will Employers Hire Former Offenders?

Employer Preferences, Background Checks, and Their Determinants.” Pages 205-243 in Mary Pattillo, David Weiman, and Bruce Western (eds.), Imprisoning America: The Social Effects of Mass Incarceration. New York: Russell Sage Foundation.

Jacobs Valentine, Erin. 2012. Returning to Work After Prison: Final Results from the Transitional Jobs

Reentry Demonstration. New York, NY: MDRC.

Kelly, B., Margolis, M., McCormack, L., LeBaron, P.A., Chowdhury, D. 2017. “What Affects People’s

Willingness to Participate in Qualitative Research? An Experimental Comparison of Five Incentives.” Field Research. Volume: 29 issue: 4, page(s): 333-350

Lerner, Debra, David A. Adler, William H. Rogers, Hong Chang, Annabel Greenhill, Elina Cymerman,

and Francisca Azocar. 2015. "A randomized clinical trial of a telephone depression intervention to

reduce employee presenteeism and absenteeism." Psychiatric Services 66, no. 6: 570-577.

Looney, Adam, and Turner, Nicholas. 2018. Work and Opportunity Before and After Incarceration. Washington, D.C.: The Brookings Institution.

Mercer, Andrew, Andrew Caporaso, David Cantor, and Reanne Townsend. 2015. “How Much Gets You Much? Monetary Incentives and Response Rates in Household Surveys.” Public Opinion Quarterly 79: 105-129.

Moffitt, Robert. 2004.  The Three-City Study Incentive Experiment: Results from the First Two Waves.  Retrieved from http://www.jhu.edu/~welfare.

Mueser, Kim T., Robert E. Drake, and Gary R. Bond. 2016. "Recent advances in supported employment for people with serious mental illness." Current opinion in psychiatry 29, no. 3: 196-201.

Pager, Devah. 2003. “The Mark of a Criminal Record.” American Journal of Sociology 108: 937-975.

Redcross, Cindy, Megan Millenky, Timothy Rudd, and Valerie Levshin. 2012. More Than a Job: Final

Results of the Center for Employment Opportunities. New York, NY: MDRC.

Singer, Eleanor, John Van Hoewyk, Nancy Gebler, Trivellore Raghunathan, and Katherine McGonagle.

“The effect of incentives in telephone and face-to-face surveys.” Journal of Official Statistics. 1999;15: 217–30.

Singer, Eleanor and Richard A. Kulka. 2002. “Paying respondents for survey participation,” in Studies of Welfare Populations: Data Collection and Research Issues, eds. Michele Ver Ploeg, Robert A. Moffitt and Constance F. Citro, National Academy Press, Washington, D.C., pp. 105-127.

Singer, Eleanor, and C. Ye. 2013. The Use and Effects of Incentives in Surveys. The Annals of the

American Academy 645, 112-141.

Uggen, Christopher, Sara Wakefield, and Bruce Western. 2005. “Work and Family Perspectives in

Reentry.” Pages 209-243 in Jeremy Travis and Christy Visher (eds.), Prisoner Reentry and Crime in America. New York: Cambridge University Press.

Wang, Philip S., Gregory E. Simon, Jerry Avorn, Francisca Azocar, Evette J. Ludman, Joyce McCulloch, Maria Z. Petukhova, and Ronald C. Kessler. 2007. "Telephone screening, outreach, and care management for depressed workers and impact on clinical and work productivity outcomes: a randomized controlled trial." Jama 298, no. 12: 1401-1411.

Wood, Robert G., Sheena McConnell, Quinn Moore, Andrew Clarkwest, and JoAnn Hsueh. 2010. The Building Strong Families Project: Strengthening Unmarried Parents’ Relationships: The Early Impacts of Building Strong Families. Washington, D.C.: U.S. Department of Health and Human Services.

1 This covers a few types of questions in the survey, including recovery from substance use disorder, mental and physical health. Some also overlap with what would fall under “economic security” such as housing situation/stability and food insecurity.

2 Please refer to Supporting Statement B of this information collection request for more detail on the team’s plans to maximize response rates and minimize non-response bias.

3 See page 69, questions 75 and 76, https://obamawhitehouse.archives.gov/sites/default/files/omb/assets/omb/inforeg/pmc_survey_guidance_2006.pdf

10