Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
TANF and Child Support Moving Forward: Lessons Learned from the COVID-19 Pandemic
Formative Data Collections for Program Support
0970-0531
Supporting Statement
Part B
September 2022
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
330 C Street, SW, 4th Floor
Washington, DC 20201
Project officers:
Lisa Zingman
B3. Design of Data Collection Instruments 5
B4. Collection of Data and Quality Control 6
B5. Response Rates and Potential Nonresponse Bias 8
B6. Production of Estimates and Projections 9
B7. Data Handling and Analysis 9
Part B
The objective of the Temporary Assistance for Needy Families (TANF) and Child Support Moving Forward: Lessons Learned from the COVID-19 Pandemic study is to document and learn from innovations that TANF and child support programs have made in response to evolving operational and service delivery needs emerging from the COVID-19 pandemic. The study will document the adaptations made by TANF and child support programs during the pandemic, especially sustained changes that are likely to continue for the foreseeable future. Virtual small group interviews with participants will be used to learn about program participants’ perceptions of and experiences with TANF and child support programs during the COVID-19 pandemic and beyond. ACF will use the information collected through this study to provide timely information to TANF and child support programs about changes and adaptions that they and other TANF and child support programs are making to improve their operations and services, as well as inform related future directions for research.
Generalizability of Results
The study is intended to present internally-valid descriptions of chosen TANF and child support sites, not to promote statistical generalization to other sites or service populations.
Appropriateness of Study Design and Methods for Planned Uses
The study’s purposive site selection approach and qualitative data collection methods are appropriate for the goal of collecting comparable information and understanding the adaptations made by TANF and child support programs during the pandemic. The information collected through this study will be used to advance knowledge among federal, state, and local program administrators and other stakeholders about operational changes made in response to the COVID-19 pandemic that can inform TANF and child support program operations and service delivery during the pandemic and beyond.
This information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.
This study will systematically obtain information from TANF and child support programs about the adaptations made during the pandemic and those that will be sustained in the foreseeable future. The project will include a purposive sample of up to 20 state TANF programs and 20 state child support programs that include a diverse range of geography, program administrative structures, policies, and processes; COVID-19 impact; and TANF and child support caseload size (i.e., the number of TANF and child support cases in a state).
TANF and child support programs in selected states represent study “sites”. Respondents will be selected using non-probability, purposive sampling to identify individuals who can provide information on the study’s key research questions.
For each of the 20 selected TANF sites and 20 child support sites, we will collect information from a mix of state-level program administrators (e.g., program directors, policy and operations division directors) through a web-based questionnaire and semi-structured interviews.
In a subset of the 20 TANF and child support sites (up to 16 TANF sites and up to 16 child support sites), we will gather additional information about local implementation experiences with COVID-19 adaptations. The sites selected for local interviews will be purposively selected to capture variation in administrative structure and program characteristics. For example, local child support sites selected for additional data collection will include sites that rely on judicial procedures for order establishment and enforcement as well as sites that rely more heavily on administrative establishment and enforcement procedures. Selection for local TANF interviews intends to include geographic differences and variation in how different sites implemented a similar/the same TANF adaptation. In these local interviews, data will be collected from a mix of local staff such as local office directors or managers, county directors, frontline staff, representatives from the court system, and contracted service providers.
In a subset of the local areas (up to 4 TANF sites and up to 4 child support sites) selected for local-level staff interviews, program participants will be recruited for small group interviews about participant experiences with COVID-19 adaptations. TANF and child support participants will not be representative of the population of individuals who receive TANF and child support services.
Sampling and Site Selection
The data collection is intended to describe how the selected TANF and child support programs adapted their program operations, policies, and activities during the pandemic. It is also intended to describe the experiences of a small sample of program participants. The data collection is not intended to produce generalizable or representative information about all TANF or child support programs or program participants.
The 20 TANF and 20 child support study sites will represent a range of characteristics including geography, program structure, COVID-19 impact, caseload size (i.e., the number of TANF and child support cases in a state), TANF policies, and child support processes. Site selection will be based on input gathered by the study team from consultations with federal and external key informants, and an environmental scan of publicly available information on TANF and child support policy and operational responses to the pandemic.
Because of the purposive selection of study sites, the sites will not be representative of all TANF and child support agencies.
Study sites will be recruited for participation through multiple methods:
Letter of support from federal office staff. The Office of Family Assistance (OFA) and Office of Child Support Enforcement (OCSE) staff will circulate a letter of support to the study sites (Appendix A: Letter of support from OFA and OCSE).
Letter of support from key professional associations. The American Public Human Services Association (APHSA) and the National Council of Child Support Directors (NCCSD) will e-mail a letter of support to study sites (Appendix B: Letter of support from APHSA/NCCSD).
Direct outreach to sites. TANF and child support agency directors will receive an e-mail from the study team (Appendix C: State TANF/Child Support Director Recruitment Email). The recruitment e-mail will describe the COVID-19 study and will include a link to and a pdf version of the web-based questionnaire. The recruitment e-mail will also include the letter of support from federal office staff (Appendix A: Letter of support from OFA and OCSE) and the project description (Appendix D: Moving Forward project description). Following the recruitment e-mail, the study team will send a scheduling e-mail to state respondents (Appendix E: State TANF/Child Support Director scheduling e-mail), thanking them for their completion of the survey and requesting to schedule a one-hour interview with them and their staff. Local- level staff from those sites selected for local interviews will receive a scheduling e-mail from the study team (Appendix F: Local TANF/Child Support Staff Invitation Email), explaining the study and inviting them to participate in a one-hour phone call to discuss their implementation and lessons learned for pandemic-related changes. This invitation email will also include the project description as an attachment.
Direct outreach to program participants. Following successful recruitment of local-level staff, we will provide staff with an email template for recruitment of program participants for small group interviews (Appendix G: Program Participant Recruitment Email). If local staff determine alternative ways of recruiting program participants, such as telephone calls or flyers, staff can use the content of the e-mail template to draft those recruitment materials. To obtain the targeted number of participants, the study team will overrecruit in each local area. In each local area, the study team will recruit 8 program participants. The e-mail, or other recruitment materials, will come from the local-level staff to the program participants directly and will explain the purpose of the conversations and invite them to join the conversation. Following a response from program participants, the study team will send a virtual meeting invitation (Appendix H: Program Participant Meeting Invitation) that will remind participants of the purpose of the conversation, provide instructions for using the video conferencing platform, specify the time and date of the call, and provide instructions about receiving the $30 gift card. The meeting invitation e-mail will be sent to the respondent immediately after the meeting is scheduled and then resent again 1 day prior to the meeting as a reminder.
Types of interview respondents, data collection instruments to be used, and the approach for respondent selection are explained below.
Web based questionnaires
State TANF agency web-based questionnaire (Instrument 1a) will be administered to up to 20 state TANF program administrators about changes in operations and service delivery initiated or accelerated by the pandemic and changes that have been sustained. The questionnaire will be sent to selected state TANF program directors. The questionnaire e-mail will explain that while multiple staff can be consulted to complete the questionnaire, only one questionnaire will be completed by each study site.
State child support agency web-based questionnaire (Instrument 1b) will be administered to up to 20 state child support program directors about changes in operations and service delivery initiated or accelerated by the pandemic and changes that have been sustained. The same procedures used for the TANF web-based questionnaire will be used for the child support questionnaire.
Semi-structured interview guides for state and local TANF and child support program staff.
State TANF and child support staff interview guide (Instrument 2a). A single interview with up to 3 state-level respondents will be conducted at each TANF and child support study site. The study lead will conduct initial outreach to the state program directors; however, directors will be given the option to include up to 2 additional state-level staff members who are knowledgeable about the program and the changes that occurred throughout the pandemic, such as policy and operations staff.
Local TANF and child support staff interview guide (Instrument 2b). Interviews with up to 3 local-level respondents will be conducted at a subset of the TANF and child support sites for a total of 32 interviews (16 with TANF sites and 16 with child support sites). Local level interviews will be held with a mix of local staff. Such staff will be interviewed in small groups, as small group interviews with local staff will provide the most efficient way of capturing multiple perspectives.
Semi-structured small group participant interview guide (Instrument 3).
TANF participant interview guide (Instrument 3a). Small group interviews will be held in up to four TANF sites with up to 15 TANF participants across all sites. The 60-minute virtual small group interviews will each include up to 5 participants at each local site to understand participant perspectives of changes made to TANF programs.
Child support program participant interview guide (Instrument 3b). Small group interviews will be held in up to four child support sites with up to 15 child support participants across all sites. The 60-minute virtual small group interviews will each include up to 5 participants at each local site to understand participant perspectives of changes made to child support programs.
B3. Design of Data Collection Instruments
Development of Data Collection Instruments
The data collection instruments were developed by content experts at Mathematica and MEF, and informed by reviewing instruments used in similar data collection efforts, including a draft survey developed by the Office of Inspector General about the effect of the COVID-19 pandemic on paternity establishment; a survey circulated by the National Council of Child Support Directors to member programs about changes to practices during the COVID-19 pandemic; and a survey circulated by the American Public Human Services Association to its members on changes made by TANF programs in response to the COVID-19 pandemic1.
The instruments were developed to capture essential data for the study’s guiding questions. The instruments were closely examined to confirm that they were streamlined and not collecting duplicative data across respondents.
The project team does not intend to conduct a pre-test of the data collection instruments. The web-based questionnaire is intended to collect background information that can be used to customize the interview guides. Each semi-structured interview guide allows questions to be tailored to the respondent based on his or her organization and role. Not all questions will be asked of all respondents, and interviewers will ensure they use the program names, acronyms, and terminology that is appropriate for each respondent to understand the questions that are asked. Prior to interviews, interviewers will learn from the program contact how to word questions to take into account program-specific titles or terminology.
B4. Collection of Data and Quality Control
Web-based questionnaire
The web-based questionnaire will be circulated to up to 20 TANF and 20 child support program state-level program directors. The web-based questionnaire will be administered via QuestionPro, an online survey questionnaire tool appropriate for collecting high level standardized information. Each program director will receive an email inviting the program to participate (Appendix C: Recruitment email from study team). This email will introduce the project and will be used to schedule a call to discuss the project and data collection activities in more detail. An attachment to the email will include a project description (Appendix D: Moving Forward Project Description). Data collection will begin with circulating the web-based questionnaire to state-level TANF program directors (Instrument 1a: State TANF agency web-based questionnaire) and child support program directors (Instrument 1b: State child support agency web-based questionnaire).
There are two versions of the web-based questionnaire—one specific to the TANF program and one specific to the child support program. It is expected that each web-based questionnaire will take up to 20 minutes to complete. Both questionnaires will include a cover sheet explaining to respondents that (1) responses to the questionnaire will be used for research purposes (2) although information collected through the questionnaire will likely be incorporated into public materials in the aggregate, information will not be attribute to questionnaire respondents, and (3) information about the respondents will not be made public. Respondents will be asked to submit completed questionnaires within 2 weeks of receipt.
Semi-structured staff interviews
Following completion of the web-based questionnaire, the study team will send a scheduling email to the state program director (Appendix E: State TANF/Child Support Director scheduling e-mail) and will schedule a one-hour meeting with state-level TANF and child support staff (Instrument 2a: State TANF and child support staff interview guide). Up to three state-level staff will be included in each interview—the program director and two additional staff, such as policy or operations staff.
Following the state-level interviews, a subset of sites will be selected for local level interviews. The study team will send an invitation email to local level respondents (Appendix F: Local TANF/Child Support Staff Invitation Email) and will then schedule and conduct up to 16 TANF and 16 child support interviews with local level staff (Instrument 2b: Local TANF and child support staff interview guide). Local level interviews will be held with a mix of staff and will include up to 3 respondents in each interview and will last about one hour.
The interviewers will offer privacy assurances as part of the introduction to the interview. With permission from respondents, interviews will be recorded to support the notes taken during the interviews. The semi-structured interviews with state-level program staff will not be administered in its entirety in each interview. Rather, for each interview, questions will be selected and that are relevant to each respondent and program. Similarly, semi-structured interviews with local-level program staff will not be administered in its entirety, as well. Questions will be selected that are relevant to each respondent and program.
Participant interviews
To facilitate small group discussions, participant interviews will each be limited to 5 participants. To successfully engage the target number of participants (15 TANF participants and 15 child support participants), the study team will identify up to 4 local TANF programs and up to 4 local child support programs to recruit program participants. The study team will provide local level staff with a recruitment e-mail to send directly to program participants (Appendix G: Program Participant Recruitment Email). After participants agree to participate in the small group interview, the study team will send a virtual meeting invitation (Appendix H: Program Participant Meeting Invitation). The meeting invitation e-mail will be sent to the respondent immediately after the meeting is scheduled and then resent again 1 day prior to the meeting as a reminder.
Semi-structured group interviews will be conducted virtually with TANF program participants (Instrument 3a: TANF participant interview guide) and with child support participants (Instrument 3b: Child support participant interview guide) for 60 minutes. Prior to the start of the interview, interviewers will ask participants for their consent to participate in the interview and to be recorded. Interviewers will inform participants that their responses will be kept private to the extent permissible by law and ask for permission to record the interview. If all participants consent to being recorded, the interviewer will start the recording and note that all participants have consented to being recorded. If participants do not provide consent to be recorded, notes will be taken in lieu of a recording. As each interview progresses, the interviewer will select and ask questions that are relevant to the participants.
Quality control
For the web-based questionnaire, all questionnaires will be reviewed for completeness. The web-based questionnaire will also include contact information for study team staff who can answer respondents’ questions about the questionnaire.
For the semi-structured interviews with staff and participants, staff conducting these interviews will participate in data collection training to ensure data are collected systematically and consistently. Topics to be discussed in the training include interviewing techniques, protecting privacy, and procedures for ensuring data security. The training will also include reviewing the data collection instruments to ensure full understanding of their purpose and the collection of comparable, complete, and high-quality data.
After the interview data are collected, the project team will input the data into a standard template across all programs. Completed templates will be reviewed to check for gaps in information or inconsistencies.
B5. Response Rates and Potential Nonresponse Bias
The study is not designed to produce representative or generalizable findings and site participation is wholly at the program’s discretion. The study sample will include up to 1) 20 TANF sites (and 5 alternate sites) and 2) 20 child support sites (and 5 alternate sites). If a site is not responsive or declines the invitation to participate, it will be replaced with an alternate site.
It is anticipated that in each site, multiple state- and local-level program staff will participate in the interviews. To maximize response rates, within selected sites, staff and participant interviews will be scheduled to accommodate respondents’ availability and scheduling needs. If there is an instance when a targeted respondent is unable to meet at the scheduled time, a member of the project team will reschedule the call at a more convenient time or request to meet with an alternate respondent in a similar position. Particularly in the case of program participants, the project team will be prepared to conduct shorter, 30-minute interviews with participants individually, to reach the desired number of participant responses.
In total, it is expected that 40 interviews will be conducted with up to 123 child support respondents and 40 interviews will be conducted with up to 123 TANF respondents.
As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated.
The study is designed to produce accurate descriptions of participating sites and does not need to secure a set number of sites or participation rate to be of value to the field.
The study team will not conduct nonresponse bias analysis or explicitly adjust for nonresponse, as it is not necessary for these purposes.
The scheduling of tele- and video-conference interviews will be flexible to accommodate the particular scheduling needs of respondents and conducting follow–up interviews if necessary. The project team will be especially flexible in scheduling interviews with program participants, including being available outside of business hours, being flexible to accommodate changes to respondents’ schedules. Program participants who participate in the semi-structured small group interviews, which are estimated to take 60 minutes on average, will receive a $30 gift card as a token of appreciation.
B6. Production of Estimates and Projections
The data collected is for internal evaluation team use only and will be used for descriptive purposes. Data will not be used to make any population estimates, either for internal use or dissemination.
B7. Data Handling and Analysis
Data Handling
All interview questions are open-ended, and notes will be taken during the small-group interviews. Interview audio files and notes will be stored in an encrypted project folder on Mathematica’s network. Mathematica uses access control lists to restrict access to the encrypted project folders where sensitive and confidential project data are stored. Access to the project folder is explicitly authorized by the Project Director on need-to-know and least privilege bases. Mathematica staff are required to change their password for computer and network access every thirty days, and passwords must adhere to strict composition standards. Staff access rights to the project folder are revoked when they leave the project. If a staff member leaves Mathematica, his or her access to computing assets, including network access, is terminated.
Data Analysis
We will analyze state-level responses to the questionnaire to learn more about each state and the key changes they made during the pandemic. We will also use the findings from the questionnaire to tailor the semi-structured interview protocols that we will use to guide our discussions with program staff and participants.
We will use summary templates to systematically summarize and organize key information collected through the questionnaire and semi-structured interviews with program staff and participants. For the program interviews, we will develop two summary templates - one that aligns with the protocols for child support staff and another that aligns with the protocols for TANF staff. For the participant interviews, we will also develop two separate summary templates to summarize the information collected from child support and TANF participants, respectively. Each summary template will summarize all the staff interviews, or all the participant interviews conducted in that state. The summary templates will overlap when possible but will be adapted accordingly to align with the information that will be collected for each program. All summary templates will also include a standardized section where we will populate key information about the state using the responses provided in the state-level questionnaire.
The project team will use the qualitative findings to report the key pandemic-related changes made by TANF and child support programs that will be sustained into the foreseeable future, the lessons programs learned from making these pandemic-related changes, and participant experiences with these changes. The project team will use NVivo to analyze the summary templates and examine the key findings from each state. The summary template will be formatted as to allow for auto-coding in NVivo using the key topics and themes of interest to the study. The project team will then do additional manual coding as needed to further analyze key topics or themes that arise during the interviews.
Data Use
The findings from this study will be incorporated into reports and other T/A resources that will be shared publicly in an effort to provide timely information to TANF and child support programs about changes and adaptions that they and other TANF and child support programs are making to improve their operations and services, and better support the well-being of families. These documents will describe the data collection and analysis process and note that the results are not generalizable to improve understanding of how to properly interpret study findings. State program directors and local office directors will be given the opportunity to review these documents to confirm their accuracy before publication.
Project leaders from the Office of Planning, Research, and Evaluation (OPRE), Mathematica, and MEF Associates who designed and will collect and analyze the data are as follows:
Lisa Zingman, Administration for Children and Families, OPRE, Federal Project Officer
Elaine Sorensen, Administration for Children and Families, OPRE, Content Expert
Pamela Holcomb, Mathematica, Project Director
Rebekah Selekman, Mathematica, Deputy Project Director
Lindsay Ochoa, Mathematica, COVID-19 Lessons Learned Co-task Lead
Bethany Boland, MEF Associates, COVID-19 Lessons Learned Co-task Lead
Attachments
Instruments
Instrument 1a State TANF agency web-based questionnaire
Instrument 1b State child support agency web-based questionnaire
Instrument 2a State TANF and child support staff interview guide
Instrument 2b Local TANF and child support staff interview guide
Instrument 3a TANF participant interview guide
Instrument 3b Child support participant interview guide
Appendices
Appendix A Example letter of support from Office of Family Assistance/Office of Child Support Enforcement
Appendix B Example letter of support from American Public Human Services Association/ National Council of Child Support Directors
Appendix C Recruitment email from study team
Appendix D Moving Forward project description
Appendix E State TANF/Child Support Director scheduling email
Appendix F Local TANF/Child Support staff invitation email
Appendix G Program participant recruitment email
Appendix H Program participant meeting invitation
1 Note that these surveys were designed and used for internal uses; they are not publicly available.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | OMB App B |
| Author | OPRE and Mathematica |
| File Modified | 0000-00-00 |
| File Created | 2023-07-31 |