BIAS-NG Generic Supporting Statement B 061517_clean

Generic Data Collection (OMB Number 0970-0XXX)

SUPPORTING STATEMENT PART B-Statistical Methods

FOR OMB CLEARANCE

Behavioral Interventions to Advance Self-Sufficiency Next Generation Project

July 2017

 

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

TABLE OF CONTENTS

1. Respondent Universe and Sampling Methods 2

2. Procedures for the Collection of Information 3

3. Methods to Maximize Response Rates and Deal with Non-response 5

4. Test of Procedures or Methods to be Undertaken 5

The proposed data collection will consist of a series of mixed-methods studies to identify, develop, and test interventions related to social service and benefit receipt in the program areas of Temporary Assistance for Needy Families (TANF) and Child Welfare. Behavioral Interventions to Advance Self-Sufficiency-Next Generation (BIAS-NG) seeks to gather information from state and local agencies and their clients and staff to inform intervention design and evaluation. Each of the proposed studies under this Information Collection will involve a focused scope and moderate-sized samples.

B1. Respondent Universe and Sampling Methods

Target sites for this series of studies consist of states, and local agencies providing services under the auspices of two ACF programs: TANF and Child Welfare. A list of potential sites is currently being identified through two primary avenues: 1) Referrals from ACF program office staff familiar with state and local human services programs; 2) Interest expressed voluntarily by staff from human services programs while attending presentations on the BIAS project at public meetings and conferences. After potential sites are identified, based on referral and/or interest, we will review available information such as marketing and recruitment materials, program manuals, and organization charts. Based on available information, the research team and ACF will select up to 6 organizations for fieldwork to ensure a mix of program areas, populations, locations, and service approaches.

The target respondents to be included in this generic information collection (IC) include but are not limited to:

• Customers or individuals receiving services from state and local ACF programs in the domains of TANF and Child Welfare.

• Staff working in state and local programs or agencies in the domains of TANF and Child Welfare.

In studies covered under this generic ICR at Phases 1-3, obtaining probability-based samples to reach the desired subpopulations of interest (e.g., foster parents or TANF clients) will be cost-prohibitive and not needed for achieving study goals. Purposeful, targeted sampling through specific programs and other non-probability sampling designs will be used to develop a pool of potential respondents, potentially drawing from state or county caseloads.

Over the 3 years this clearance covers (covering Phases 3 and 4) we anticipate meeting with up to 72 administrators, 144 frontline staff, and 144 clients in either interviews or focus groups across six sites. Additionally we plan to field surveys to up to 6,600 clients and 240 staff across these same 6 sites over the course of the two phases. The review of marketing and recruitment materials, program manuals, and organization charts will help determine the administrative staff or clients to include in focus groups or interviews. There will be an estimated total of 2,400 participants (including administrator, staff, and client interviews, focus groups, and surveys) each year and a total of 7,200 respondents over the 3 years that this clearance covers.

Because the Phase 1 through 3 samples are not randomly selected, they may be biased and not fully represent the entire study population. We will use purposive sampling to select potential participants for interviews and focus groups. At early stages, interviews and focus groups will not be used as data collection, but to inform intervention design.

Once sites have been identified and interventions have been designed, there will be subsequent data collection for both the implementation and the impact studies undertaken during the evaluation stage, Phase 4. For the implementation studies, this data collection will involve more formal interviews and/or focus groups. The implementation studies will also include surveys, which could use either random or purposive sampling, depending on the availability of information about the sampling frame, time, and resources. The limitations associated with purposive or any sampling method will be described in the Phase 4 submission, and will be clearly stated in any publications produced for this project. For the impact study, data collection will rely on administrative and/or MIS data, as described below.

Universe of Data Collection Efforts

Data collection activities at Phases 3 and 4 include:

• Administrator interviews/focus groups: In order to diagnose the problem, the research team plans to gather data during Phase 3 to better understand the barriers administrators see to full program implementation. In Phase 4, the research team plans to return to interview program administrators to determine whether the intervention was administered with fidelity and to help to determine the intervention’s effect on program administrators.

• Staff interviews/focus groups: Collecting information from program staff during Phase 3 will help the researchers better understand how the program operates from the staff perspective and what barriers staff see to the program operating at its potential. Staff interviews/focus groups may incorporate prototyping activities.¹ Returning to staff during Phase 4 will shed light on what aspects of the intervention worked well and which didn’t work well from a staff perspective.

• Client interviews/focus groups: Interviewing clients during Phase 3 will help researchers better understand the barriers clients face when trying to access and interact with the program. Client interviews/focus groups may incorporate prototyping activities. These insights will help inform the interventions targeted at clients. Talking with clients during Phase 4 will help researchers better understand from a client perspective what is, or is not working with the intervention.

• Client survey: Surveying clients can provide researchers with a more representative sample of client opinions as to how the program operates both before (Phase 3) and after the intervention (Phase 4).

• Staff survey: Surveying program staff can provide researchers with a more representative sample of staff opinions as to how the program operates both before (Phase 3) and after the intervention (Phase 4).

• Administrative data: The research team will work with sites to access administrative data the agencies are already collecting in the format in which the site collects it. This administrative data will allow us to track proximal outcomes such as attendance at required appointments, submission of completed paperwork, or referral to additional services. This will not involve any burden on staff or clients.

• Direct observation: The research team plans to be a “fly on the wall,” observing staff and client interactions. This observation will not involve any burden on staff or clients.

• Reviewing Case Files: As part of interviewing staff, the research team may ask staff to share de-identified case files of their clients so that we can better understand the needs of the clients they serve, how they document interactions, and the type of follow-up they engage in. This type of data collection will not impose burden because the team will not ask for the information to be provided in a specific format other than the one in which it already exists.

• MIS data collected: If a site collects client data in an MIS system, the research team would like to obtain, with no burden to the staff, demographic and study-specific data at the time of enrollment into the study (i.e., random assignment data) and tracking data in order to see implementation measures for our sample. This data will be requested in the existing format used by the site.

• Collecting site documents: If site staff send written materials to clients, the research team may request copies of these materials, at no additional burden to the staff.

Analysis Plan – Phase 4

Impact Analysis: The collection of the administrative data from each site will allow us to conduct impact analysis of each intervention. We plan to use a factorial design for our impact evaluation. The sample sizes required for impact studies are based on the assumptions that most tests will use either a standard two group design or a 2x2 factorial design and the main outcome will be binary (i.e., percentage). This design provides an optimal balance between the complexity of the hypotheses that can be tested and the interpretability of the results. The standard statistical tests in this design are for main effects and an interaction. Main effects test the impact of one variable averaging across the levels of the other. The power for these tests is determined by the overall sample size of the study. Since we do not yet know the full set of interventions that will be undertaken, we provide power calculations that show the minimum sample size required to detect statistically significant true impacts with 80 percent likelihood.² Detailed plans for each study site will be included in Phase 4 submissions.

Table 1: Sample Size Estimates for Tests of Main Effects in 2x2 Factorial Designs and Simple Two-Group Design RCT

Notes: Calculations assume a two-tailed test, significance level of 10 percent, 80 percent power and equal allocation of participants to levels. Estimates are based on a binary outcome (such as using a service); minimum detectable effects are percentage point increase from baseline. Continuous outcomes require additional assumptions.

Implementation Analysis: The collection of qualitative data from each site will allow us to conduct implementation analysis of each intervention. The implementation study will describe and document each site’s intervention, how it operated, and provide information about the contrast in treatment between the research groups—both whether the planned contrast between the treatment and the control condition occurred (implementation fidelity) as well as how the treatment implemented actually differed from the status quo (treatment contrast). This information will be important for interpreting the findings of the impact study. The implementation study will result in lessons for the field about how the interventions operate, the challenges they face, and the participants’ (clients and/or staff) perspectives on whether the behavioral interventions changed their behavior. Although specific components of the implementation study will depend upon the sites’ specific behavioral interventions, our plan relies on a mixed-methods methodology, employing both qualitative and quantitative approaches.

Cost Analysis: The BIAS-NG project will include a cost analysis for all sites. While we could conduct a benefit-cost analysis, it would be important to have long-term follow-up for key outcome measures to monetize benefits. Thus, we will need to determine whether such an analysis is possible on a site-by-site basis. The cost analysis will estimate the per person cost of the intervention(s) over and above what is spent on the control group.

B2. Procedures for the Collection of Information

To gather information and inform intervention design, study teams composed of at least two members will make the telephone and in-person meeting contacts. These staff members are experienced in the process of gathering information for purposes of designing demonstration and evaluation projects.

The study team will send each agency’s program director information about the study and offer the opportunity to speak with members of the study team. An overview email will be addressed to program directors, when relevant, and will introduce the study, its goals, and the team executing the proposed study on ACF’s behalf. Tailored emails will be included within individual generic ICRs. Attached to the email will be the project description. The study team will be available to answer any questions about the study. When relevant for identifying a potential match between the study and a site, we may ask for select programmatic information, such as their administrative structure, experience, target population, and size, when relevant. We will cover a set of topics relevant to the study and specific to the site to allow us to understand the variation of programming in the field, the range of perspectives on the BIAS-NG study, and whether particular study design options will be feasible given the structure of the agencies’ programs. With a select group of programs, the study team will follow-up on any initial conversations with a request for further individualized discussion and data review to gain a better understanding of the program’s flow and solicit feedback about the potential interventions and study designs. Following initial analysis of these data to understand the flow of participants, the study team will conduct an in-person visit to select sites. Subsequent visits and teleconferences will be scheduled, as needed and with a narrower pool of programs, if the study team needs additional time to gather the information.

Once sites have been selected, we plan to conduct in-person focus groups and phone interviews to help ensure an effective design for the intervention. There are three separate protocols: one protocol to use for interviews and focus groups with staff who deliver services; one protocol to use for interviews and focus groups with administrators; and one protocol to use for interviews and focus groups with clients. The protocols in Appendix A provide an outline for the basic procedures that will be used for each data collection approach (i.e., focus group or individual interviews), the types of questions that will be asked and the expected flow of questions. Once finalized, instruments tailored to individual sites will be submitted through individual ICRs under this generic ICR.

Focus groups will be facilitated by at least two individuals; one individual will conduct the in-person interviews and surveys. Each focus group/interview will begin with an introduction that explains the purpose and goals of the BIAS-NG project. Participants will then be asked to read and sign the consent form. The facilitator will inform participants that the conversation will be audio-recorded but that they will be able to have any comments they don’t want repeated removed from the recording. Lastly, the Paperwork Reduction Act will be explained and the OMB number for this collection and the expiration date will be provided. In the focus groups, once all participants have signed consent forms, the facilitator will ask each participant to introduce themselves and then begin the conversation. For individual in-person and phone interviews, the facilitator will simply begin the conversation with the participant after receiving consent. At the end of the interview, participating clients will receive a gift card worth up to $20. Specific information about proposed incentives will be included in individual generic information collection requests under this clearance.

The focus groups and interviews in Phase 3 are designed to be formative and exploratory. Human services program staff and clients possess procedural and tacit knowledge that will be vital for identifying areas where behavioral insights may have a high impact. We plan to spend a maximum of 60 minutes with each staff person during each site visit. This data collection will be used only for descriptive purposes, not as part of an impact evaluation.

For subsequent data collection for implementation research to better understand how well interventions have been implemented, at Phase 4, focus groups and interviews will be structured. Not all questions will be asked of each respondent, based on the participant’s background or experience. We will reduce burden by asking only relevant questions. For descriptive or implementation research purposes, the study may field self-administered questionnaires, either online or in person, to agency or program staff. In a mixed-mode approach, the survey firm first attempts to survey each respondent by telephone. Field interviewers then attempt to interview respondents who cannot be contacted by telephone in person. Study participants can refuse to complete the survey, or refuse to answer any of the questions on the survey, and will not be penalized in any way. Examples of items and instruments are provided in Appendix A; once they are developed and prior to use in the field, tailored, site-specific instruments will be submitted with additional information as generic IC requests for each of the approximately 12 tests across the six sites, along with information about the associated intervention.

B3. Methods to Maximize Response Rates and Deal with Non-response

At this formative stage to inform intervention design and site selection, for focus groups and interviews, we will take several steps to help ensure a high rate of cooperation among respondents. First, ACF federal staff has pre-existing collaborative relationships with many program sites. We will leverage these relationships to help secure buy-in from site staff to both participate in, and aid deployment of, data collection tools. The ultimate aim of the BIAS-NG project is to provide program sites with lessons to improve their on-the-ground operations. Explaining to sites the benefits they will receive from this project may help persuade sites that their effort is worthwhile.

For subsequent data collection from selected sites, the research team will also work closely with administrators and staff to develop recruitment strategies for clients and program staff for focus groups and interviews, particularly to make sure we gather a group that reflects a mix of experiences. As is usually the case with focus groups, we will recruit at least double the number of people for each focus group with the anticipation that half will not attend.

Staff working with the program and control groups will be asked to complete a survey several months after the launch of the intervention. Based on the response rates for the staff surveys in the Enhanced Transitional Jobs Demonstration, we expect around 85 percent of staff to complete the survey. For surveys, we will use established methods, such as sending reminders, setting completion deadlines, attempting to reach participants by phone after several failed attempts to obtain a response, and working closely with staff to maintain an accurate list of contact information.

To further increase the likelihood of participation, we will also offer clients participating in focus groups, interviews, and in-person surveys incentives, as discussed in Supporting Statement Part A.

B4. Tests of Procedures or Methods to be Undertaken

No pre-testing has been completed at this time and there are no plans to pretest focus group or individual interview protocols. It is possible that formative focus groups and interviews may inform the development of focus group and interview protocols for subsequent data collection and inform the development of survey instruments. We may pre-test surveys, if it is necessary to develop novel items or instruments for the project, with 9 or fewer staff from a similar program that will not be in the study.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Kim Clum

Social Science Research Analyst

Office of Planning, Research, and Evaluation

Administration for Children and Families

330 C St. SW, 4th Floor

Washington, DC 20201

(202) 205-4922

Victoria Berk Kabak

Social Science Research Analyst

Office of Planning, Research & Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

330 C St. SW, 4th Floor

Washington, DC 20201

(202) 401-5871

1 Prototyping involves showing two versions of materials to people, observing how they interact with each, and asking them to explain their reactions to each. Questions start out general and become more specific if there are particular words or phrases within the materials that the designer wants to focus on. Respondents may be asked whether they understand the program’s rules and what they are being asked to do, what aspects they do not understand, and if they have suggestions for changes to the materials.

2 Since power calculations to determine the required sample size for a factorial design are conducted essentially the same way as in a standard two-group RCT (Somers et al, 2014), following Bloom (1995), we use , the formula to calculate the minimum detectable effect to determine the sample size for specified MDEs. We do this because we will not know the baseline outcomes (π) from which our study will be based until the Program Area Domains and problems are selected. Solving [1] for n, yields:

[2]

Where, = the proportion of the study population that would have a value of 1 for the binary outcome in the absence of the program

= the proportion of the study sample that is randomly assigned to the treatment group

= minimal detectable effect, which is smallest true impact that an experiment has a good chance of detecting

M = a multiplier equal to 2.49, representing the statistical significance level of 0.10 and power of 0.80.

6