BIAS-NG Generic Supporting Statement A 082917_clean
BIAS-NG Generic Supporting Statement A 082917_clean.docx
ACF Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) Project
OMB: 0970-0502
Generic Data Collection (OMB Number XXXX-XXXX)
FOR OMB CLEARANCE
ACF Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) Project
July 2017
Submitted By:
Office of Planning, Research and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
A1. Necessity for the Data Collection
The Office of Planning, Research and Evaluation (OPRE) at the Administration for Children and Families (ACF), U.S. Department of Health and Human Services (HHS), seeks Office of Management and Budget (OMB) approval for a 3-year pilot generic clearance to conduct interviews, focus groups, and surveys with state and local Temporary Assistance for Needy Families (TANF) and Child Welfare program administrators, staff, and clients in order to inform the design of and to better understand the mechanisms and effects of interventions informed by behavioral science and intended to improve program outcomes. This activity is planned as part of ACF’s Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) project. The BIAS-NG project will apply behavioral insights to a range of ACF programs including Temporary Assistance for Needy Families (TANF), Child Welfare, and other programs to be determined and will use behavioral insights to design and test interventions intended to improve the operations and efficacy of human services programs. The request is being sought as a pilot of the concept of using a generic approval mechanism for the types of studies described in this document. As such, studies included under this clearance will be limited to two program areas: TANF and Child Welfare.
|
Study Background
The September 2015 Executive Order “Using Behavioral Insights to Better Serve the American People” stated that “A growing body of evidence demonstrates that behavioral science insights -- research findings from fields such as behavioral economics and psychology about how people make decisions and act on them -- can be used to design government policies to better serve the American people” and encouraged federal agencies to “develop strategies for applying behavioral science insights to programs and, where possible, rigorously test and evaluate the impact of these insights.” In keeping with this directive, OPRE plans to conduct the Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) project. This project will use behavioral insights to design and test interventions intended to improve the operations and efficacy of human services programs. The BIAS-NG project builds on a prior OPRE project, the Behavioral Interventions to Advance Self-Sufficiency (BIAS) project, which relied exclusively on administrative data to test the short-term impact of small “nudge” interventions in human services programs. The BIAS-NG project will build on and go beyond the BIAS project by applying behavioral insights to additional ACF programs, going beyond testing simple “nudges” to include helping programs be more self-reflective about how they present choices and options to participants, testing alternative approaches to presenting those options and, importantly, by collecting qualitative information from program staff and participants to better understand the mechanisms and effects of behavioral interventions. Information collected from interviews, focus groups, and surveys with program staff and participants will first enable the research team to better diagnose problems amenable for behavioral interventions. Based on this information, the research team will be able to design relevant interventions. Information collected during the implementation of the interventions will provide additional information as to whether the intervention was successful and, just as importantly, why or why not.
OPRE is requesting generic clearance to conduct these interviews, focus groups, and surveys over a period of three years. The BIAS-NG study is designed such that each specific intervention is designed in consultation with the agency leaders; the timeframes are shorter than many evaluations because outcomes of interest are proximate to the intervention point; and these studies often lend themselves to rapid cycle evaluation where testing a particular intervention design can inform subsequent tests of related program improvement efforts.
The iterative and rapid nature of these tests poses a challenge to complying with the timeline for seeking full approval of each individual information collection activity subject to the Paperwork Reduction Act (PRA). Thus, OPRE requests generic clearance to conduct this work. For each specific information collection under this generic approval, instruments will be tailored to the specific intervention and the specific site; once a set of instruments for a particular test is developed, and prior to use in the field, OPRE will submit a supporting statement Part A and B and submit the specific instruments to be used to OMB for approval. Each specific information collection will include two submissions: first, a submission for the formative stage research, to include supporting statements (Stage 3 in Exhibit 1 below); and second, a submission for the test and evaluation materials, to include supporting statements (Stage 4 in Exhibit 1 below).
Legal or Administrative Requirements that Necessitate the Collection
There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.
A2. Purpose of Survey and Data Collection Procedures
Overview of Purpose and Approach
Generic Information Collections (ICs) submitted under this control number will consist of the following criteria:
A full Supporting Statement A and Supporting Statement B will accompany each of the Gen ICs submitted under this generic clearance. These will include:
A discussion of the respondents. Administrators, staff, and clients will be the subjects of our research during this IC.
Information about the context of each specific IC. Researchers will speak with and survey specific populations in a particular geographic location/setting/agency.
A description of the planned qualitative data collection which will include submission of the specific instruments for review. Instruments will include focus group/interview protocols and short surveys specific to each informant group (agency administrators, staff, and clients).
A description of the qualitative analyses planned. Audio recordings and notes from interviews/focus groups will be analyzed for patterns and themes.
A description of the administrative data that the agencies are already collecting and that the project will utilize. It is important to note that collecting administrative data will not impose a burden on respondents or record keepers, as we intend to ask sites to provide data as it currently exists. We will not be requesting that it be provided in any particular format that is different from the format in which the agency typically keeps it.
A description of the planned intervention associated with each specific IC.
Information about planned communication about the findings. Study outcomes will be communicated to state and national stakeholders in a position to consider and implement site-specific improvements to ACF agency programs.
Final proposed instruments will accompany each of the Gen ICs submitted under this generic clearance.
Any supplementary materials (advance letters, emails, etc.) will accompany each of the Gen ICs submitted under this generic clearance, as appropriate.
The study is designed to develop tools to apply behavioral insights to ACF human services programs, design and test interventions informed by behavioral science, encourage rapid cycle tests that may lead to further improvements in human services programs, and enable state and local program staff to learn skills to engage in behavioral diagnosis and design, and conduct rigorous test of future interventions. The interventions we design for this study will address problems that have broad relevance for TANF and Child Welfare programs. While it is our intention for the specific findings from each intervention to provide information that could be useful in the design and operation of programs that provide similar services to similar populations, the specific findings from these interventions will only be suggestive and preliminary, based on this research. The limitations of such findings will be made clear in any related communications.
The majority of the work in each site will be conducted in five phases. Exhibit 1 provides an overview of the process to take place in each site, which consists of planning phases to determine the program area domains and learn about the problems of interest to stakeholders (Phase 1) and identify sites (Phase 2). Phase 3 is where we expect to engage with administrators, program staff, and clients through interviews (via telephone or in-person) and/or focus groups. These interactions are needed to develop the interventions to test. During Phase 4 we plan to conduct implementation research with sites, interviewing administrators, program staff, and clients to better understand how the test is being implemented. The below bullets provide more detail on the work that will occur during each phase.
Planning Phases
Phase 1 (late 2015 – mid 2017):
Select 4 Program Area Domains
Two of the four domains will be TANF and Child Welfare. These domains were pre-selected by ACF and are included under this request for clearance.
We may identify up to 2 additional domains in consultation with ACF and federal leadership of program areas (2017), which would be included in a separate Federal Register publication to allow for at least 30 days for public comment on the additional domain areas.
Define the Problem Areas in Each Domain
To ensure that our pilot interventions do not address problems idiosyncratic to a particular program, we identified a set of problems that broadly affect TANF and Child Welfare programs.
Phase 2 (mid 2016 –2019):
Identify 6 Sites
Identify up to 6 sites across the two already-identified domains. Interest in participating in BIAS NG is expected to be high and it is not expected that systematic recruitment of sites will be necessary.
Generic Information Collection Phases
Phase 3: Diagnose 6 Sites and Design 12 Tests (early 2017 – 2019)
Conduct behavioral diagnosis and design at each of the 6 sites
Behavioral diagnosis and design is a procedure in which we examine the process related to the problem of interest (to better understand the factors that may be inhibiting the desired outcomes and design solutions that are informed by behavioral science research to help improve outcomes. For example, through this process we can identify barriers that TANF recipients may face that contribute to their lack of engagement in welfare-to-work programs.)
During this phase we plan to review preexisting individual level, de-identified administrative data from each site and complete the first round of interviews/focus groups and surveys included under this clearance in order to best identify the bottlenecks and when and how an intervention would be the most useful.
Phase 4: Conduct 12 Evaluation Tests (mid 2017 – 2019)
Conduct evaluation of the designed intervention
The mixed methods evaluation will consist of implementation, impact, and cost research
The implementation study will rely in part on the second round of interviews/focus groups and surveys included under this clearance.
Dissemination Phase
Phase 5: Disseminate Findings and Archive Data (early 2018 – 2019)
Write 12 test-specific briefs
In addition to collecting data from administrators, staff, and clients with focus groups, interviews, and surveys, we also intend to supplement this information with administrative data the agencies are already collecting. Collecting administrative data will not impose a burden on respondents or record keepers, as we intend to ask sites to provide data as it currently exists. We will not be requesting that it be provided in any particular format that is different from the format in which the agency typically keeps it. In addition, we will not be asking more than nine individuals to provide the administrative data.
Research Questions
For the purposes of designing the intervention and conducting an evaluation of its implementation, we will conduct interviews, focus groups, and surveys with administrators, staff, and clients. These qualitative data collection activities are critical to designing an effective intervention, allowing the research team to properly diagnose ways in which agencies are not maximizing their impact for the populations they serve. These activities will allow the team to gather structured in-depth information to understand the program process from both the administrative and client perspectives. Focus groups and interviews are essential to identifying the points in the outreach and delivery of services, or in the client’s experiences, that are most amenable to a behavioral intervention. They will allow the BIAS-NG team to map a correspondence between the insights of behavioral science with the on-the-ground implementation of programs and subsequent client experiences.
These qualitative data collection activities are also essential to conducting implementation research, to describe and document each site’s intervention, how it operated, and provide information about the contrast in treatment between the research groups – both whether the planned contrast between the treatment and control condition occurred (implementation fidelity) as well as how the treatment implemented actually differed from the status quo (implementation contrast). This information will be critical to interpreting the findings of our interventions.
Please see Attachments A.1-A.5 for sample interview, focus group, and survey questions. Once sites are selected and instruments are tailored for each site, and for both Phase 3 and Phase 4, we will submit individual IC requests with additional detail about the site, the final tailored instruments, and the site-specific study methodology.
Study Design
Phase 3: Diagnosis and Design
During Phase 3, we will collect qualitative data from administrators, staff, and clients via focus groups, interviews, and surveys which will help to inform our intervention design. Changes to instruments used by the federal study team will be submitted to OMB for approval. We will also collect administrative data from agency MIS systems to better understand client experiences with the program and identify points where service delivery might need improvement.
Phase 4: Evaluation Tests
Impact Study
During Phase 4, we will design and conduct impact analyses of behavioral interventions. Such interventions may include but are not limited to:
participant reminders, such as emails, text messages, or telephone calls to facilitate the completion of a particular action;
implementation prompts, which encourage participants to make a plan for when they are going to complete an action;
easy tracking tools for clients to make it simpler for them to show they are meeting program requirements;
self-affirmation exercises to counter individuals’ tendency not to complete an action if they perceive it as a threat to their self-conception or identity;
restructured work flows and processes to improve service delivery;
automatic enrollment, which defaults eligible participants into a program so that they must opt out rather than opt in;
pre-population of forms to make it easier and faster for clients to complete lengthy or confusing forms; and
co-location of services to reduce the barriers associated with traveling to multiple offices for different benefits.
It is possible that, in conjunction with some of the behavioral interventions, sites may decide to change what data they collect and/or the questions they ask the public to answer. Such decisions will be controlled by the sites not the project. Our proposed framework of selecting sites within pre-specified Program Domains – TANF and child welfare – and targeting similar problems across these sites could also provide opportunities for replication and to determine if similar interventions are effective in different settings trying to get to the same outcomes. When appropriate, we may use factorial or sequential study designs to assess the effectiveness of each intervention component with the goal of building the most efficient intervention possible.
Implementation Study
Additionally in Phase 4, we will also conduct an implementation study to describe and document each site’s intervention, how it operated, and provide information about the contrast in treatment between the research groups—both whether the planned contrast between the treatment and the control condition occurred (implementation fidelity) as well as how the treatment implemented actually differed from the status quo (treatment contrast). This information will be important for interpreting the findings of the impact study. Exhibit 2 presents research questions that will be addressed by information collection in Phase 4. Changes to instruments used by the federal study team will be submitted to OMB for approval. Phase 4 will also include a cost analysis.
Exhibit 2: Research Question and Instrument Matrix
Research Questions |
Administrator interviews/focus groups |
Staff interviews/focus groups |
Client interviews/focus groups |
Client survey |
Staff Survey |
How are sample members identified and recruited for the intervention? |
X |
X |
|
|
X |
To what extent were the interventions implemented with fidelity? |
X |
X |
X |
X |
X |
For example, what are the patterns of participation (if appropriate as a proximal measure) and do these patterns adhere to the intervention design? |
X |
X |
|
|
X |
What were the challenges and barriers the site experienced? |
|
X |
|
|
X |
How did the system within which the program operates influence implementation? |
X |
X |
|
|
X |
What is the organizational culture and how does it support or hinder responses to the behavioral intervention? |
X |
X |
|
|
X |
To what extent did the intervention require collaboration between multiple agencies or units, and what worked well and what did not? |
X |
X |
|
|
X |
What are the participant perspectives on their response to the intervention? |
|
|
X |
X |
|
A3. Improved Information Technology to Reduce Burden
Planning site visits will be done collaboratively with each of the sites. We will use conference calls and emails to the extent possible to minimize burden.
The interviews will be conducted either individually or as a focus group. To minimize the burden, we will hold semi-structured group discussions (focus groups), rather than individual conversations, whenever possible. Each group discussion will include staff at the same or similar levels. For example, one group discussion may be held with multiple front-line workers, such as case workers or outreach specialists. A separate group discussion may be held with supervisors of front-line staff. A third discussion group may include staff at the management or administrative level, such as directors of offices or agencies. If there is a single staff member in a particular level, however, an individual discussion will be held. We anticipate that staff at each of these levels will have different perspectives and thus may have different experiences. Group discussions will allow us to reduce the length of time spent at the site while still obtaining valuable feedback on the planning grants from staff with a range of experiences.
The surveys will be administered on the web.
A4. Efforts to Identify Duplication
The information collection requirements for this study have been carefully reviewed to determine what information is already available from existing studies and program documents and what will need to be collected for the first time. Although information from existing sources improves our understanding of the planning process, ACF does not believe that it provides sufficient information on how TANF and Child Welfare agencies interact with their clients. This data collection is intended to yield new and useful information about TANF and Child Welfare processes. The interviews and focus groups will support a deeper exploration of patterns seen in the survey data or review of documents.
A5. Involvement of Small Organizations
We do not anticipate any small organizations to be affected by these information collections. Nonetheless, we will schedule interviews at times that are convenient to participants in order to minimize disruption of daily activities.
A6. Consequences of Less Frequent Data Collection
Rigorous evaluation of innovative initiatives is crucial to building evidence of what works and how best to allocate scarce government resources. These data collection undertakings represent an important opportunity for ACF to both learn about activities associated with TANF and Child Welfare, and also design behavioral interventions to improve service delivery and uptake.
Not collecting information from the three categories of respondents (administrators, staff, and clients) during Phase 3 would limit the government’s ability to design appropriately targeted interventions that appropriately match the barriers administrators, staff, and clients face in the quest for optimal service delivery. Not collecting information during Phase 4 would hinder the government’s ability to learn how interventions were implemented and whether and to what degree the interventions had the outcome desired.
A7. Special Circumstances
There are no special circumstances for this data collection.
A8. Federal Register Notice and Consultation
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on May 23, 2017, Volume 82, Number 98, page 23572, and provided a 60-day period for public comment. A copy of this notice is included as Attachment 1. No substantive comments were received during the notice and comment period.
Consultation with Experts Outside of the Study
We may consult with relevant stakeholders and experts on the study design and data collection instruments. When needed, specific consultants will be identified in each Generic IC.
A9. Incentives for Respondents
In order to make sure we have a range of experiences represented by our respondents, we plan to offer clients participating in focus groups, interviews, and in-person surveys a gift card worth up to $20. Our hope is that a gift card ‘thank you’ will help offset the financial burden that may result from extenuating travel, additional cell-phone data or phone minutes, or child care costs associated with in-person focus groups.
Incentives will not be used as a substitute for other best-practice persuasion strategies designed to increase participation, such as explanatory advance letters, endorsements by people or organizations important to the population being surveyed, and assurances of privacy.
We will include a written justification in the specific generic IC request for any tokens of appreciation and we will secure Institutional Review Boards (IRB) approval for the use and monetary value of the use of incentives prior to fielding the survey and hosting focus groups. Additional information will be provided in each individual generic ICR.
A10. Privacy of Respondents
All respondents who participate in research under this clearance will be read a statement that will explain the study and will inform individuals that their participation is voluntary and of the extent of their privacy as respondents. (See Attachments A.1-A.5) Participants will be told verbally that their conversations will not be shared in a form that identifies them with anyone outside the research team. As ACF’s prime contractor, MDRC plans to implement all data collection activities. If data collection activities are performed by a subcontractor, that subcontractor will maintain the same standards of privacy as required by MDRC. Information will be kept private to the extent permitted by law and in accordance with current federal information security standards and other applicable regulations.
MDRC employees are required to maintain and process quantitative and qualitative data in designated project folders on the MDRC network. With the exception of the temporary storage of data during onsite collection, MDRC employees are not allowed to download, keep, or process individual-level data on the hard drives of their MDRC work stations or any other storage. Information will not be maintained in a paper or electronic system from which they are actually or directly retrieved by an individuals’ personal identifier.
The project Data Manager will organize BIAS NG project folders and will supervise storage of BIAS NG data files on a “need-to-know” basis. Following standard MDRC practice, the project Data Manager and project programmers will replace all PII from incoming source data with a randomly-generated project ID number. Also these files will be saved in secure folders with limited access on a “need-to-know” basis. Thereafter, most data processing for the project will be performed on analysis files that have been stripped of PII. All reports, tables and printed materials are limited to presentation of aggregate numbers. MDRC will destroy all paper records and electronic records containing PII when no longer needed for research purposes in accordance with funder and contractual requirements, as well as MDRC retention policies.
A11. Sensitive Questions
There are no sensitive questions in this data collection.
A12. Estimation of Information Collection Burden
Exhibits 3 provide details about how this estimate of burden hours and costs were calculated.
During the Diagnosis and Design Phase (Phase 3), we anticipate meeting with approximately 4 administrators per site, at 2 sites per year for each of the 3 years, for a total of 24 people. We anticipate meeting with up to 8 frontline staff per site, at 2 sites per year, for each of the 3 years, for a total of 48 people. We plan to meet with up to 8 clients per site, at 2 sites per year, for each of the 3 years, for a total of 48 people. We plan to administer surveys to up to 100 clients per site, at two sites per year, for each of the 3 years for a total of 600 clients. We anticipate administering surveys to up to 20 staff members at 2 sites for each of the 3 years for a total of 120 staff surveys.
During the Evaluation Phase (Phase 4), we anticipate meeting with approximately twice the number of respondents per category as in Phase 3 (48 administrators, 96 staff, and 96 clients). We plan to survey ten times the number of clients as in Phase 3, 1,000 clients at each of the 6 sites (6,000 total clients). We anticipate surveying the same number of staff as in Phase 3, 20 staff at each of the 6 sites (120 total staff). These are the number of people we intend to extend the survey to, not the number of people who actually respond (we will strive for the 80 percent response rate standard). As discussed in Part B, we will endeavor to reduce burden on individual respondents by asking only relevant questions. Accordingly, we think that the estimate below represents an upper bound on potential burden.
We calculated the overall burden per respondent by multiplying the frequency of response by the time to complete each data collection item. We anticipate that focus groups for administrators, staff and clients (Attachments A.1-A.5) will each take 1 hour to complete. We anticipate the client and staff surveys to each take approximately 15 minutes to complete online. Each respondent will not be tasked more than once over the course of the study. The information collection for both phases is specific to each site and is not intended to continue once the study is over.
Exhibit 3: Burden Hours
Instrument |
Total Number of Respondents |
Number of Responses Per Respondent |
Average Burden Hours Per Response |
Total Burden Hours |
Average Hourly Wage |
Total Cost |
PHASE 3: DIAGNOSIS AND DESIGN |
||||||
Administrator interviews/ focus groups |
24 |
1 |
1 |
24 |
$33.90 |
$813.60 |
Staff interviews/ focus groups |
48 |
1 |
1 |
48 |
$33.90 |
$1,627.20 |
Client interviews/focus groups
|
48 |
1 |
1 |
48 |
$15.55 |
$746.40 |
Client survey |
600 |
1 |
.25 |
150 |
$15.55 |
$2332.50 |
Staff Survey |
120 |
1 |
.25 |
30 |
$33.90 |
$1,017.00 |
PHASE 4: EVALUATION |
||||||
Administrator interviews/focus groups |
48 |
1 |
1 |
48 |
$33.90 |
$1,627.20 |
Staff interviews/focus groups |
96 |
1 |
1 |
96 |
$33.90 |
$3,254.40 |
Client interviews/focus groups |
96 |
1 |
1 |
96 |
$15.55 |
$1,492.80 |
Client Survey |
6,000 |
1 |
.25 |
1,500 |
$15.55 |
$23,325.00 |
Staff survey |
120 |
1 |
.25 |
30 |
$33.90 |
$1,017.00 |
Total |
7,200 |
|
|
2,070 |
|
$37,253.10 |
Total Cost
To compute the total estimated cost for staff and administrators, the total burden hours were multiplied by $33.90, the mean hourly earnings for management, professional and related workers in the civilian workforce as reported by BLS NCS (2010).1 To compute the total estimated annual cost for clients, the total burden hours were multiplied by $15.50, the mean hourly earnings for high school graduates reported by BLS NCS (2010). The estimated total cost is $37,253.10.
A13.Cost Burden to Respondents or Record Keepers
There are no additional costs to respondents.
A14. Estimate of Cost to the Federal Government
The total cost for the data collection activities under this current request will be approximately $3,708,213. Annual costs to the Federal government will be approximately $1,236,071.
A15. Change in Burden.
This is a new data collection.
A16. Plan and Time Schedule for Information Collection, Tabulation and Publication
Time Schedule and Publication
Exhibit 4: Generic IC and Publications Time Schedule
|
CY 2017 |
CY 2018 |
CY 2019 |
CY2020 |
||||||||||
|
Q3 |
Q4 |
Q1 |
Q2 |
Q3 |
Q4 |
Q1 |
Q2 |
Q3 |
Q4 |
Q3 |
Q4 |
||
PHASE 3: Diagnosis and Design |
Diagnosis and Design |
|
|
|||||||||||
PHASE 4: Evaluation |
|
|
Evaluation |
|||||||||||
PHASE 5: Dissemination |
|
|
|
|
Dissemination |
|||||||||
Phase 3: Diagnosis and Design: This phase involves the development of site-specific diagnosis and design of behavioral intervention(s) and an evaluation plan using a collaborative process with the site, behavioral science and program content experts, and ACF staff. During this time period we will undertake Phase 3 for six total sites.
Phase 4: Evaluation: Phase 4 consists of implementing the behavioral intervention(s) and evaluating them. During this time period we will undertake Phase 4 for six total sites, with two tests per site, for a total of 12 tests.
Phase 5: Dissemination: Dissemination efforts during the time of this clearance includes site specific reports, infographics, dissemination products aimed at practitioners, sharing findings at conferences, and publicizing our findings and our work on social media.
A17. Reasons Not to Display OMB Expiration Date
All instruments will display the expiration date for OMB approval.
A18. Exceptions to Certification for Paperwork Reduction Act Submissions
No exceptions are necessary for this information collection.
1 U. S. Department of Labor, Bureau of Labor Statistics “National Compensation Survey: Table 1: Summary Mean hourly earnings and weekly hours for selected workers and establishment characteristics.” 2010. http://www.bls.gov/ncs/ocs/sp/nctb1344.pdf.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |
© 2024 OMB.report | Privacy Policy