Supporting Statement B

Status: Original and derived artifacts are available for this document.

Download: docx | pdf | html

Supporting Pilot Program Outcomes, Research and Technical Assistance in TANF (Project SUPPORTT) Implementation and Outcomes Studies



OMB Information Collection Request
New Collection




Supporting Statement
Part B




May 2026


Type of Request: New


Submitted By:
Administration for Children and Families 
U.S. Department of Health and Human Services

3rd Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201

Project Officers:
Lauren Deutsch Stanton (COR)  
Amelia Popham (ACOR)  


Part B

B1.	Objectives
Study Objectives
The  Administration for Children and Families (ACF) seeks approval for data collection activities related to conducting the implementation and outcomes studies for the Supporting Pilot Program Outcomes, Research, and Technical Assistance in TANF project (Project SUPPORTT). The Fiscal Responsibility Act of 20231 (FRA) requires the U.S. Department of Health and Human Services (HHS) to carry out a pilot program for the Temporary Assistance to Needy Families (TANF) program under which it may select up to five states to be held accountable to outcomes-based performance measures instead of the Work Participation Rate (WPR). ACF is administering this TANF Pilot program. The legislation also requires a report to Congress on the pilot and its outcomes. 

ACF awarded the Project SUPPORTT contract to (1) provide research, evaluation, and data-related technical assistance (TA) to help pilot states meet the requirements of the Pilot, including negotiating new TANF performance benchmarks for work and family outcomes; (2) facilitate peer learning and other opportunities for pilot and non-pilot states to interact and learn from each other; and (3) design and conduct implementation and outcome studies of the pilot programs to inform a report to Congress as required by P.L. 118-5. The objectives of the implementation study are to describe pilot design, staffing, service provision, partnerships, contextual details, and factors that support or hinder the pilot’s implementation. The objective of the outcomes study is to assess changes in participant outcomes using surveys and administrative data. 

Generalizability of Results 
The outcomes study is intended to produce internally valid estimates of participant outcomes in pilot states, not to promote statistical generalization to other sites or service populations. The implementation study is intended to present internally valid descriptions of the TANF pilot programs and their implementation in the five pilot states, not to promote statistical generalization to other sites or service populations. 

Appropriateness of Study Design and Methods for Planned Uses 
Conducting outcomes and implementation studies of the five selected pilot states is appropriate for the government’s goal of describing what pilot states do in response to the FRA legislation and to examine the outputs and outcomes of the pilots. The results of the outcomes and implementation studies will be used to inform a legislatively required report to Congress on the TANF pilots.
 
Outcomes study. The study will use the most rigorous methods feasible to understand the outcomes of the TANF pilots on TANF participants. Given that it is not possible to randomly assign participants to pilot conditions, the study will employ a quasi‑experimental design (QED) comparing outcomes for two groups of TANF participants: individuals who exited TANF prior to implementation of the pilots and individuals who enrolled during the pilot period. The analysis will include statistical controls to construct a comparison group that is as similar as possible to the treatment group. Although QEDs do not ensure full baseline equivalence between groups, careful design and analytic techniques can reduce bias and strengthen causal inference, allowing observed outcome differences to be reasonably associated with the pilots while acknowledging the limitations relative to randomized designs.
 
Implementation study. Findings from the implementation study will be used to document and understand how states responded to the FRA policy change and to support interpretation of the outcomes study findings. The implementation study will describe the design of the pilots; how states implemented the pilots and factors that help or inhibit states’ progress toward quality implementation; and changes to TANF pilot programs’ outcomes during the pilot period, such as changes in the allocation of resources, staff time use, and services offered.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information. 

B2.	Methods and Design
Target Population  
The study team will collect information from TANF program staff, partner staff, and TANF participants in the five pilot states. For the outcomes study, the study team will survey the two groups of TANF participants described in B1: individuals who exited TANF prior to implementation of the pilots and individuals who enrolled during the pilot period. Eligible respondents will be TANF participants who completed the Study information form (OMB #0970‑0356) and agreed to be contacted for the 12-month post-TANF-exit survey (Instrument 7).

The target population for the implementation study will be TANF staff and participants who have participated in the program during key periods of the pilot to be able to answer questions about their experiences with delivering or receiving services. Because respondents will be purposively selected, they will not be representative of the population of TANF staff, partner staff, and TANF participants. Instead, the study team aims to obtain variation in staff and participant experiences to understand engagement with the pilot services and processes within a given state.

Sampling and Site Selection
Sampling for outcomes study. The sample frame for the outcomes study will be TANF participants who complete the Study information form (OMB #0970-0356). The study team anticipates that approximately 400 participants per group per pilot state (approximately 800 total participants per state) will complete the Study information form and become the starting sample for the 12‑month post‑TANF‑exit survey. These figures represent expected sample numbers rather than fixed targets. The Study information form serves solely as the mechanism for identifying and informing eligible TANF participants about the 12-month post-TANF-exit survey (Instrument 7). Respondents will be invited and reminded to participate in the 12-month post-TANF-exit survey through email, text message, and letters sent through the mail (Appendix A). 
 
Table B.1 reports program-level minimum detectable impacts on earnings outcomes for survey and administrative data. The expected sample size for each state is 800 study participants (approximately 400 from each of the groups described in B1), or 4,000 for up to five states. 
 
The minimum detectable impacts of a survey sample of 800 is about $142 in average monthly earnings. This means that the study is powered to detect a difference of $142 or greater in monthly earnings between the pre-pilot and the pilot groups as measured through the follow-up surveys. If the true difference in earnings between the groups is less than $142, the study will likely not detect a statistically significant impact. The minimum detectable impact corresponds to a minimum detectable effect size of 0.19. Evidence reviews, such as the What Works Clearinghouse, consider effect sizes of 0.25 standard deviations or larger as substantively important2, which the study will be able to detect. However, note that these are only estimates based on data from previous studies on similar populations.  

Table B.1. Minimum detectable impacts per program on key outcomes for a QED 
Response rate 
Analysis sample size 
(treatment and comparison) 
Monthly earnings (impact, dollars) 
Effect size
100 percent 
800 
142 
0.19 
80 percent 
640 
159 
0.22 
70 percent
560
170
0.23
Assumptions: 50 percent of the sample will be from the pre-pilot group and 50 percent will be from the pilot group; $736 standard deviation of earnings; covariates explain 10 percent of the variation in the outcomes; two-tailed test, p-value of 0.05. 

Sampling for implementation study. The implementation study will use a combination of purposive (nonprobability) sampling and census approaches. 

For data collection activities aimed at obtaining in-depth, contextual information—such as the data collected using the Pilot staff discussion guide, Pilot costs and resources workbook, and In-depth participant interview guide—the study team will use purposive sampling to select respondents based on their roles, responsibilities, and experiences with the TANF pilot. TANF program leaders, staff, and relevant partner staff will be identified using organizational information and their level of involvement in program implementation, and participants will be selected to reflect a range of experiences within each state for example, either directly with the implementation of the pilot or through roles that are affected by the pilot such as staff that oversee contracts or data. Recruitment and scheduling materials in Appendix B will be used for the Pilot staff discussion guide, outreach materials in Appendix D will be used for the Pilot costs and resources workbook, and recruitment and scheduling materials in Appendix E will be used for the In-depth participant interview guide. This approach is appropriate because the information sought depends on respondents’ specific experiences with the pilot and roles.

For data collection activities intended to capture information from a broad group of staff, including the data collected via the Leadership survey, Staff survey, and Staff time use survey, the study team will use a census approach by inviting all TANF program and partner staff involved in delivering employment and case management services in the pilot states to participate, using notifications in Appendix C. These activities will provide quantitative information across a wide range of staff involved in program implementation.

Site selection. Within each pilot state, the study team will purposively select local TANF offices for inclusion in implementation study activities based on state-specific criteria such as the number of work-eligible individuals, urbanicity, and level of participation in the pilot. This approach will ensure that selected sites reflect variation in program context and implementation.

B3.	Design of Data Collection Instruments
Development of Data Collection Instruments
The data collection instruments were developed to capture essential data to achieve the objectives described in B1 and address the study’s main research questions in A2. This information is not readily available from administrative sources.  

Data for outcomes studies. The outcomes study survey instrument, the 12-month post-TANF-exit survey (Instrument 7), was developed by content experts at Mathematica and ACF and informed by reviewing instruments used in similar OMB-approved data collection efforts. Many questions are sourced from prior studies, such as the Parents and Children Together study (OMB #0970-0403), Evaluation of Employment Coaching for TANF and Related Populations (OMB #0970-0506), and the Next Generation of Enhanced Employment Strategies Project (OMB #0970-0545). Other items come from scales that have been frequently used in large-scale national surveys, such as the K-6 Distress Scale to measure levels of psychological distress3. 
 
The study team used industry best practices to reduce potential sources of measurement error. These practices include: 
    • Using validated items from previous surveys administered to similar populations to the extent possible. 
    • Including in the instruments automatically enforced skip patterns, built-in range checks, internal item consistency checks, and required answer fields.  
    • Pretesting the 12-month post-TANF-exit survey with individuals similar to the populations served by the type of programs being assessed for inclusion in the TANF Pilot. The project team timed the surveys and used cognitive interviewing and respondent and interviewer debriefings to assess respondents’ understanding of the survey questions, identify improvements to the flow and structure of the instruments, and to ensure burden was low. The same questions were posed to fewer than ten people so this activity did not require PRA clearance. As a result of the pretest, the surveys were updated for clarity, flow, and to reduce burden.  
 
Data for implementation studies. The implementation study instruments were developed by content experts at Mathematica, The Adjacent Possible, and ACF. They were informed by reviewing instruments used in similar OMB-approved data collection efforts. These efforts included the Evaluation of Employment Coaching for TANF and Related Populations (OMB #0970-0506), the Next Generation of Enhanced Employment Strategies Project (OMB #0970-0545), and TANF Case Studies (OMB #0970-0533). 
 
The study team pretested the staff and leadership surveys (Instruments 2 and 3) and the time use survey (Instrument 4) with staff and leaders from two of the pilot states. The study team also pretested the interview guide for in-depth participant interviews with TANF participants from the pilot states. The same questions were posed to fewer than ten people so this activity did not require PRA clearance. As a result of the pretests, the surveys and interview guide were updated for clarity, flow, and to reduce burden.  
  
B4. 	Collection of Data and Quality Control
Outcomes study. The study team will collect data for the outcomes study via a survey of former TANF participants (as described in B2 and B3) and administrative records. The 12-month post-TANF-exit survey (Instrument 7) will be available to participants to self-administer via the web. The survey will also be available to participants to complete via Computer Assisted Telephone Interviewing (CATI) and in-person.

The study team will ensure quality and consistency in the data collected through the survey by using different tactics, such as: 
    • For self-administered web-based surveys: provide clear instructions; use clear and straightforward language; include predominantly closed-ended questions; include check boxes, drop-down menus, and response categories; include data checks to prevent outlier entries and inconsistent responses; and ensure the layout is compatible with multiple browsers, tablets, and smartphones. 
    • Provide a study Help Desk accessible by phone or email that respondents can use if they have questions while completing the surveys. 
    • Recruit qualified interviewers to administer the survey over the phone and field locators to locate participants and connect them to a phone interviewer to complete the survey.
    • Train the phone and field interviewers in interviewing techniques as well as the intent of each question in the survey.
    • Listen to 10 percent of all interviews completed with interviewers at the study team’s call center for any inaccurate presentation of information on the study; errors in reading questions; biased probes; inappropriate use of feedback in responding to questions; and any other unacceptable interviewer behavior.
    • Examine data on the number of completed phone interviews, calls made, refusals, refusal conversions as well as time per call, and time per interview by interviewer. Supervisors will provide feedback to phone interviewers based on this data. 
    • Debrief with groups of phone and field interviewers shortly after the start of a data collection to discuss the respondents’ level of cooperation and ability to understand and answer the survey questions. 
    • Examine frequencies and cross-tabulations on data collected on a regular basis to pinpoint any unexpected aspects of instrumentation, particularly in skip logic, valid value ranges, the operation of edits and consistency checks, and the recording of data for legitimately skipped items and “don’t know” and refusal responses. 
    • Examine frequencies and cross-tabulations on data collected, by mode of collection, to look for evidence of mode bias or large differences in responses between self-administered web-based surveys and interviewer-administered telephone interviews.

Implementation study. The implementation study will collect data using multiple instruments, including the Pilot staff discussion guide (Instrument 1), Leadership survey (Instrument 2), Staff survey (Instrument 3), Staff time use survey (Instrument 4), Pilot costs and resources workbook (Instrument 5), and In-depth participant interview guide (Instrument 6). Data collection will occur at key points during the pilot period, including pilot planning and start-up, early implementation, and steady-state implementation, with some instruments administered more frequently.

To obtain in-depth, contextual information, the study team will conduct discussions with TANF program leaders, staff, and partner staff using the Pilot staff discussion guide and conduct interviews with participants using the In-depth participant interview guide. These discussions and interviews will occur during site visits and at selected points throughout the pilot period and will focus on program design, implementation, and participant experiences. Interviewers will be trained on how to conduct interviews using these instruments. Data collected from interviews will be periodically reviewed for quality and completeness.

To collect quantitative information from a broad group of staff, the study team will administer the Leadership survey, Staff survey, and Staff time use survey via the web. The Leadership survey and Staff survey will be administered at key stages of pilot implementation. The Staff time use survey will be administered every six months. The surveys will use structured response options and programmed checks to promote data quality, and the study team will monitor responses for item nonresponse and other indicators of data quality.

To document program costs and resource allocation, the study team will collect data using the Pilot costs and resources workbook at multiple timepoints during the pilot period. The study team will review available financial records and request additional information as needed to complete a standardized workbook, working with state and local leaders to ensure accuracy and completeness.
 
B5.	Response Rates and Potential Nonresponse Bias
Response Rates
    • 12-month post-TANF-exit survey. The study team anticipates that 70 percent of the TANF recipients who responded to the approved Study information form (OMB # 0970-0356) and consent to participate in the 12-month post-TANF-exit survey will complete the survey. This estimate is informed by the team’s experience conducting similar studies with hard-to-engage populations. For example, in the Youth At-Risk of Homelessness (YARH) study (OMB # 0970-0574), the team achieved a 70 percent response rate on a 12-month follow-up survey with a similarly vulnerable population.  Other recent studies have achieved higher response rates, such as Employment Coaching for TANF and Related Populations (OMB #0970-0506), which achieved a response rate of 76 percent. However, the design of these studies included more than one follow-up survey that served as additional touchpoints and anchors to the study as well as interactions with program staff who encouraged survey completion. Therefore, the expected response rate has been adjusted accordingly. To reach the anticipated 70 percent response rate, the study team plans to offer multiple modes for survey completion, including an option to complete it via phone and in-person field outreach. The study team also plans to utilize different methods to notify respondents, including email and text message. The project team does not anticipate significant item nonresponse on the follow-up survey based on prior experience asking similar questions with similar populations. To maximize participation, the study team will offer a $75 token of appreciation.
    • Discussions with program leaders, staff, and partners. The interviews are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported. To maximize participation, the study team will begin scheduling with TANF program staff and partner staff weeks in advance to ensure the timing is convenient. 
    • Leadership surveys, staff surveys, and staff time use surveys. Based on similar research projects, the study team expects program leaders will support staff members’ completion of the surveys, resulting in a high response rate among staff and leaders (90 percent). On the Evaluation of Employment Coaching for TANF and Related Populations (OMB #0970-0506), the team achieved a 90 percent response rate to a web-based survey administered to TANF staff, which used the same mode and targeted a similar population as planned for Project SUPPORTT. These surveys are designed to be easy to complete, use straightforward language, and allow respondents to break off and complete later if they get interrupted. To maximize response rates and data reliability, periodic email reminders will be sent to respondents, beginning two weeks after the data collection period begins (Appendix C). If staff and leaders do not complete their surveys within one week of the targeted time frame, the designated study team liaison will follow up with the site point of contact to remind staff and leaders that survey responses are due. 
    • Pilot costs and resources workbook. The cost and resources workbooks are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported. To maximize responses, the study team will be flexible with the data collection approach. This includes asking program leaders and/or staff to share their existing accounting records so the study team can fill in some of the information in advance and tailoring the sections of the cost workbook to only those that apply. The study team will also provide technical assistance to the leaders and/or staff as they complete the workbook.
    • In-depth participant interviews. The interviews are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported. To maximize participation, the study team will offer a $60 token of appreciation, and be flexible in scheduling interview times and locations to accommodate study participants’ schedules and needs. 

NonResponse
Outcomes study. Findings from the outcomes study are not meant to be representative, however comparability of groups is central to the QED design. The study team will follow best practices for non-response by actively monitoring response rates during survey fielding for the 12-month post-TANF-exit survey. After data collection ends, the study team will test for differences in characteristics from the Study information form (OMB # 0970-0356). The regression models described in B7 will control for any observed differences between the characteristics of treatment and comparison group respondents. The study team will  use further statistical methods to improve compatibility, such as analysis weights. 
 
Implementation study. As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Respondent demographics will be documented and reported in written materials associated with the data collection to provide context for the findings. 

B6.   Production of Estimates and Projections 
The estimates from this project will be released publicly following ACF review. The study team will also prepare a data archiving plan and archive files to support secondary analysis as described in A10.
 
Outcomes study. The outcomes study will estimate changes in participant outcomes after the shift from the Work Participation Rate (WPR) to outcomes-based performance measures (see A1). While the estimates will provide insights into the effects of this shift, the analysis will not be able to rule out that other factors could have contributed to observed differences. While statistical approaches will try to limit this concern, all estimates will be interpreted with caution. 
 
The study team will use the constructed sample weights described in B5 in the outcomes analysis so that the weighted characteristics of respondents in the comparison groups are similar to those of the full sample (respondents and nonrespondents).  
 
Survey data will be used to describe the study participants in each pilot program. The study team will use chi-squared tests of differences in means over all sample characteristics to assess whether the comparison groups have similar characteristics. The study team will also report t-tests of differences for individual characteristics. 
 
Changes in participant outcomes will be estimated for each pilot state. The study team will use regression analysis to construct more efficient estimators than the simple difference-in-means estimators, as described in B7. We plan to estimate regressions at the participant level.   
 
Implementation study. The data will not be used to generate population estimates, either for internal use or dissemination. The implementation study will describe how the pilots were designed and implemented, the state-specific context, challenges the states faced implementing the pilot and how they were addressed, changes to resource allocation associated with the pilots, changes to the program services participants received, and the pilot’s potential sustainability. This information will help interpret the outcomes study findings and provide information on the pilot process. 

B7. Data Handling and Analysis
Data Handling
Survey data. All surveys will be designed to incorporate logic rules, checks, and skip patterns to ensure data quality and accuracy as described in B3 and B4. All CATI interviewers are subject to real-time monitoring to ensure they are correctly interpreting and entering respondent responses. After data collection ends, the study team will conduct comprehensive data reviews and quality assurance reviews to ensure skip patterns are enforced and data are complete and within expected ranges.  
 
During data processing and coding, the study team will conduct quality assurance reviews to ensure consistency and minimize any data processing errors. Specifically, coders will participate in a comprehensive training session, and the project team will monitor their work, perform quality control checks, and conduct quality assurance reviews of all weighting and imputation procedures. Any outliers, skip logic errors, or other re-codes of survey data will be recorded in both internal programs and data editing spreadsheets. 
     
Interview data from discussion guides for staff and in-depth participant interviews. Study team staff will be trained on instruments for the implementation study and how to appropriately code that information after collecting it. During data coding, the implementation study task leader or other senior member of the study team will perform quality assurance reviews  to ensure consistency. 

Cost and resources data. Study team staff will be trained on completing the cost and resources workbook and how to appropriately confirm and clean information after collection. Quality assurance reviews will be conducted by the implementation study task leader or other senior member of the study team to ensure consistency. 

Data Analysis
Outcomes study. The outcomes will estimate statistical models of differences in outcomes for the pre-pilot group and pilot group. In all models, the study team will control for differences in attributes at the time of TANF program enrollment as well as other characteristics that may be associated with outcomes of interest. This approach will strengthen the quasi-experimental design. Participant demographic characteristics will be used to support analytic adjustment and interpretation of estimated impacts. For example, for participant-level analyses we plan to include controls for age, race and ethnicity, and educational attainment of the participant. 
 
The study team will also plan to conduct subgroup analysis to examine whether changes in outcomes are uniform across different groups of people participating in the TANF program. Before conducting subgroup analysis, the study team will assess whether subgroups of interest are of sufficient size to identify meaningful effects. For this analysis, the study team will identify key subgroups of interest that reflect ACF’s learning priorities. These subgroups of interest may be participants living in rural areas or participants in certain age groups or participants who pursue jobs in industries of interest to ACF. This analysis will provide insights about whether certain groups of people benefit more from the shift to outcomes-based accountability or from specific pilot program policies. Once the study team identifies the subgroups of interest, we will then estimate separate regressions for each of the identified subgroups. 
 
Implementation study. To analyze the large amount of interview data collected from multiple sources efficiently and accurately, the study team will develop a coding scheme that maps to the Consolidated Framework for Implementation Research (CFIR),4 research questions, and states’ theories of change. After coding the data, the study team will look for common themes across data elements and respondents. The analysis will include an assessment of conditions needed to replicate and sustain the pilot changes. To analyze the information, the study team will compute descriptive statistics, such as means and frequencies.  
 
Data Use
Data will be used to develop a final comprehensive report on outcomes of the pilots for submission to Congress, as required by the FRA. The study team will publish findings from the project throughout the study period in technical reports and briefs. Reporting on early implementation study findings will occur in 2027. Reporting on final implementation study and outcomes study findings will occur in 2032. In addition to presenting findings, reports and briefs will also document the methodologies used to collect, process, and analyze the project’s data across both studies; this will help the public assess study quality and interpret the findings. Study limitations and information about the generalizability of the results will be included when presenting findings. 
 
In addition, the study team will prepare final data files and documentation to be available publicly so other researchers are able to duplicate all analyses. The provided documentation will improve the understanding of how to properly interpret, analyze, and evaluate the information resulting from the data collection. The study team anticipates that data archives (restricted or public use) would become available in 2031 and hosted on a data archive platform such as the Inter-university Consortium for Political and Social Research (ICPSR).  

B8. Contact Persons
Contact information for people who can answer questions about the statistical aspects of the survey: 
    • Lauren Deutsch Stanton: [email protected] 
    • Quinn Moore: [email protected] 
 
Mathematica developed the plans for this data collection. Leaders of the project team from ACF, Mathematica, and The Adjacent Possible who designed and/or will collect and analyze the data are as follows: 
    • Lauren Deutsch Stanton, senior social science research analyst, ACF 
    • Amelia Popham, welfare research team lead, ACF 
    • Kim Clum, senior social science research analyst, ACF
    • Siri Warkentien, social science research analyst, ACF 
    • Quinn Moore, principal researcher, Mathematica 
    • Michelle Derr, CEO, The Adjacent Possible
    • Julia Lyskawa, senior researcher, Mathematica 
    • Gretchen Kirby, senior fellow, The Adjacent Possible 
    • Katie Eddins, senior researcher, Mathematica 
    • Jennifer Herard-Tsiagbey, senior survey researcher, Mathematica 
 
Attachments
    • Instrument 1: Pilot staff discussion guide 
    • Instrument 2: Leadership survey 
    • Instrument 3: Staff survey 
    • Instrument 4: Staff time use survey 
    • Instrument 5: Pilot costs and resources workbook 
    • Instrument 6: In-depth participant interview guide 
    • Instrument 7: 12-month post-TANF-exit survey 
 
    • Appendix A: Notifications for 12-month post-TANF-exit survey 
    • Appendix B: Recruitment and scheduling materials for pilot staff discussions 
    • Appendix C: Notifications for leadership, staff, and time use surveys 
    • Appendix D: Outreach materials for pilot costs and resources workbook 
    • Appendix E: Recruitment and scheduling materials for in-depth participant interviews