SSB - Generic for ACF Prog Support - SPARK

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Supporting Partnerships to Advance Research and Knowledge

Formative Data Collections for Program Support

0970 - 0531

Supporting Statement

Part B

December 2020

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers: Victoria Kabak

Part B

B1. Objectives

Study Objectives

The purpose of Supporting Partnerships to Advance Research and Knowledge (Project SPARK) is to support state and local TANF programs in implementing, replicating, and scaling interventions that promote employment and self-sufficiency for low-income families. As a part of this purpose, Project SPARK, conducted by Mathematica for ACF, has been providing technical assistance (TA) to state and local TANF programs, who are now faced with adjusting to drastic changes in their work brought on by the COVID-19 pandemic.

The purposes of the information collection are twofold: (1) to inform ACF about ways that state and local TANF programs are responding to COVID-19 and the successes and challenges they are experiencing as programs innovate, and (2) to collect feedback about how programs perceive the value of TA provided under Project SPARK. ACF will use the resultant information to inform its TA to TANF programs around continuing to provide safe and quality services to participants during the COVID-19 pandemic, and TANF programs will use the information to learn about potential strategies they can employ to maintain program operations and support participants and staff.

Generalizability of Results

This study is intended to present internally valid descriptions of the state and local TANF programs that have volunteered to receive TA under Project SPARK. It is not intended to promote statistical generalization to other sites or service populations.

Appropriateness of Study Design and Methods for Planned Uses

Through earlier individualized TA activities under Project SPARK, ACF’s contractor, Mathematica learned about myriad challenges TANF programs have been facing during the COVID-19 pandemic. The proposed information collection is designed to capture more structured and detailed information about those challenges as well as how programs have attempted to address them and their perspectives on the value of any TA around these issues they have received to date. Short surveys are an appropriate method for gathering data consistently from many programs in a manner that is both efficient and minimally burdensome to programs. ACF and Mathematica are interested in improving the overall TA provided to TANF and related programs under Project SPARK based on our learnings from this information gathering effort. We may also share summary information with programs that have received TA.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.  Limitations are described in Section A.2.

B2. Methods and Design

Target Population

The target population includes program managers, supervisors, team leads, case managers, and program analysts. These staff are from a range of different types of programs, including state and local government agencies, quasi-governmental organizations (such as Workforce Investment Boards), contracted TANF services providers, and educational institutions (such as community colleges). Because participating programs and staff have self-selected into receiving TA under Project SPARK, they will not be representative of all programs serving TANF customers.

The unit of analysis for the study is as follows:

• We will aggregate individual responses at the state and staff levels, as well as for the overall group of respondents.

As noted in Supporting Statement A, we have 310 possible respondents. We will not aggregate subgroups if the number of respondents is less than 5 so that individual respondents cannot be indirectly identified by the data.

Respondent Recruitment and Site Selection

Information collection for Project SPARK is voluntary. The project team identified TANF programs that participated in TA activities with Mathematica under four ACF contracts (OPRE’s Advancing Welfare Research and Self-Sufficiency 1 and 2, Project SPARK, and the Office of Family Assistance’s Rapid Cycle Evaluation Training and Technical Assistance project) as well as other contracts directly between Mathematica and state/county TANF agencies.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

Mathematica designed the data collection instruments specifically for Project SPARK. They are informed by the project team’s applied experience in developing, using, and refining the Learn, Innovate, Improve (LI²) framework with TANF programs (Derr et al. 2017). The LI² approach prioritizes rapid, low-burden feedback to understand programs’ experiences in real time and generate actionable insights for improvement. As a result, each survey has been designed for completion in about 10 minutes. The project team reviewed the instruments to ensure that they ask only questions necessary to achieve the objectives of the information collection.

Instruments 1-7 were developed to collect information on a range of topics pertinent to how TANF programs have had to adapt during the COVID-19 crisis. Overall, they focus on staff and customer safety and well-being; adaptations to a remote, work environment; and TA and related changes to performance measurement and accountability.

The instruments do not include scales or items that measure constructs, so identifying and minimizing measurement error are not relevant for this study design. The instruments also do not require psychometric testing. Table 1 shows how each instrument informs each objective of the information collection.

Guiding question	Instrument
	1	2	3	4	5	6	7
What are the circumstances of programs and the families they serve during the COVID-19 crisis, and how have they evolved over the course of the crisis?	X		X		X	X
How have programs responded to COVID-19? What strategies have programs developed, and what strategies show the most promise?	X	X		X	X	X	X
How has the technical assistance helped programs respond to COVID-19? What methods and topics have been most and least helpful? What are the lessons for providing remote technical assistance that might carry over into a post-COVID-19 service environment?				X			X

B4. Collection of Data and Quality Control

Instruments 1-7 will each be administered one time. A link to each survey will be emailed to TANF program staff at sites receiving COVID-related TA from Mathematica. The survey will be open for one week. For example, Instrument 1 will be administered during Week 1 of data collection. Instrument 2 will be administered during Week 2, Instrument 3 during Week 3, and so on. Respondents will be invited to reply to the surveys on a voluntary basis.

The mode of the data collection is a SurveyMonkey web survey, an intuitive, low-burden platform, deployed by Mathematica. Mathematica staff overseeing the deployment of these surveys are well-versed in the use of SurveyMonkey and will not require additional training. Mathematica’s quality assurance standards will be followed to ensure, for example, that each instrument has been appropriately programmed into SurveyMonkey and is properly deployed. To enhance data quality, we have limited the number of open response questions in favor of multiple choice and matrix-style questions. Limiting open response questions will increase the consistency of answers from different respondents.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The surveys are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion.

We will calculate response rates for overall surveys and individual questions to contextualize findings. The denominator for overall survey response rate will be the number of recipients who receive the survey. The denominator for individual question response rates will be the number of respondents who opened the survey.

Non-Response

As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. We will track the number of responses to the instruments and qualitatively report on them in any publication associated with the data collection.

B6. Production of Estimates and Projections

The data will not be used to generate population estimates, either for internal use or dissemination.

B7. Data Handling and Analysis

Data Handling

Once a survey closes, we will export the data to an Excel file and review it to confirm that the export was completed successfully, for example, by checking that the types of responses match given options for a question. We will review open response fields, such as when a respondent selects “Other” for a multiple-choice question and fills in their own answer, to determine whether the answer should be back-coded to one of the given options.

Data Analysis

The project team will generate descriptive information from survey questions, such as frequencies, means, and modal responses. For open-response questions, we will also use standard qualitative techniques, such as theme identification, to identify patterns and themes in responses.

Data Use

Findings will be discussed internally between the contractor and ACF. The findings will increase the government’s knowledge of how state and local TANF programs are responding to the COVID-19 crisis and help improve the government’s support of and TA to TANF programs. Findings may also be shared generally with TANF programs to inform their thinking about innovative practices and solutions to the challenges posed by the pandemic.

B8. Contact Person(s)

Michelle Derr

Project Director, Project SPARK

Senior Fellow, Mathematica

[email protected]

Attachments

Appendix A: Instrument 1 – Safety

Appendix B: Instrument 2 – Approaches to serving customers

Appendix C: Instrument 3 – General wellbeing

Appendix D: Instrument 4 – Engaging with coworkers remotely

Appendix E: Instrument 5 – Remote services for customers

Appendix F: Instrument 6 – Access to and use of technology

Appendix G: Instrument 7 – Performance and accountability

Appendix H: IRB Determination

Appendix I: Proposed cover email to respondents

References

Derr, Michelle, Ann Person, and Jonathan McCay. “Learn, Innovate, Improve (LI²): Enhancing Programs

and Improving Lives.” OPRE Report: 2017-108. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation. Available at https://www.acf.hhs.gov/opre/resource/learn-innovate-improve-li2-enhancing-programs-and-improving-lives.

7