Supporting Statement B

The objectives of the Temporary Assistance for Needy Families (TANF) and Child Support Moving Forward: Further Incorporating Family Input study are to: (1) understand how TANF, child support, and other human services programs gather and use family input for program improvement; (2) identify ways for TANF and child support programs to incorporate family input for program improvement; and (3) identify the successes and challenges these programs face in gathering and using family input.

To address the first and second objectives, the study previously conducted a scan of academic and nonacademic literature about how human services programs engage families in program improvement processes and held consultations with eight human services program leaders experienced in seeking and using families’ input to improve their programs. Informed by these activities, and in collaboration with an Expert Workgroup, the study developed a toolkit to support and encourage engaging families in program improvement. The Family Input Resources toolkit includes original and existing resources, guides, and tools designed specifically for administrators and practitioners of TANF and child support programs.

To address the second and third objectives, the study will pilot test the Family Input Resources toolkit with four TANF and four child support programs as they engage families in program improvement efforts of their choosing. The study will document how the pilot sites engaged families in program improvement processes; how sites used the Family Input Resources to support engaging families and improving programs; and to what extent sites found the resources actionable, relevant, and valuable to their program improvement work.

ACF will use the information collected through this study to provide timely information—through the study-informed revisions and updates to the Family Input Resources toolkit, a report, and other technical assistance resources—to TANF and child support programs to help them incorporate family input into their program improvement efforts. ACF will also use the information to inform future training and technical assistance and other program supports.

Generalizability of Results

The study is intended to present internally-valid descriptions of selected TANF and child support programs, not to promote statistical generalization to other programs or service populations.

Appropriateness of Study Design and Methods for Planned Uses

The site selection and data collection methods are appropriate for pilot testing how TANF and child support programs can gather and use family input for program improvement purposes. The information collected through this study will be used to advance knowledge among federal, state, and local program administrators and other stakeholders that can inform TANF and child support program operations, service delivery, and technical assistance needs moving forward.

This information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.

B2. Methods and Design

Target Population

The target populations for this study are program staff from TANF and child support programs run by states, tribes, territories, and counties. Selected TANF and child support programs represent study “sites.” Program site staff will pilot test the Family Input Resources toolkit. During the 12-month pilot test period, pilot site staff will lead a program improvement effort using the Family Input Resources toolkit and input from the families they serve to guide their effort.

Respondents will be selected using non-probability, purposive sampling. Each pilot site, with support from the study team, will identify up to eight staff members to serve on its “Implementation Team”. All respondents for this study are members of these site-specific Implementation Teams. Implementation Team members will represent a mix of program administrators, managers, and frontline staff. Similarly, each pilot site, with support from the study team, will identify an Implementation Team lead, who will be engaged in and lead most aspects of this work.

Site Selection

The data collection is intended to describe how the pilot sites gathered and used family input for program improvement during the pilot period. As noted above, the data collection is not intended to produce generalizable or representative information about all TANF or child support programs. Instead, the study aims to select sites that will have varied experiences using family input to inform program improvement.

Programs will self-nominate using a web-based form (Instrument 1. Expression of Interest form) to indicate interest and willingness to participate in the study. The study team will then select four TANF sites and four child support sites based on a number of factors including type (TANF vs. child support), geography (urban, suburban, rural), size, administration type (state, Tribal, or local TANF or child support program), and involvement with other Federal initiatives or research (the study team will give preference to programs that have not had extensive involvement in other Federal technical assistance or research activities). The study team will also hold semi-structured interviews with prospective sites to learn about their interests and readiness, and further explain the study commitment (Instrument 2. Prospective site conversation guide), to inform site selection. When sites are selected, two additional TANF and two additional child support sites will be selected as alternate pilot sites. Alternate sites will be invited to participate if selected pilot sites become unable to continue their participation, but no more than eight sites will participate at a time.

Study sites will be recruited for participation through multiple methods:

Email of support from federal office staff. The Office of Family Assistance (OFA) and Office of Child Support Enforcement (OCSE) staff will broadly circulate an email of support for the study to the agencies they work with (Appendix A. Family Input OFA-OCSE and APHSA-NCCSD Support Email Examples) with the study description and flier (Appendix B. Family Input Project Description and Pilot Testing Flier) as attachments. The email and flier will include a hyperlink to the expression of interest form (Instrument 1. Expression of Interest form), which prospective study sites can fill out to express interest in participating in the study. OFA and OCSE can make minor edits to tailor the text of the email, as appropriate.
Email of support from key professional associations. Select child support and TANF professional associations (for example, the American Public Human Services Association and the National Council of Child Support Directors) will be invited to send an email of support for the study to their networks (Appendix A. Family Input OFA-OCSE and APHSA-NCCSD Support Email Examples). The email of support will also include the study description and flier (Appendix B. Family Input Project Description and Pilot Testing Flier)) as attachments. The email and flier will include a hyperlink to the expression of interest form (Instrument 1. Expression of Interest form), which prospective study sites can fill out to express interest in participating in the study.
Direct outreach email to sites. The study team will supplement the broad outreach described above with direct outreach through email (Appendix C. Family Input Example Direct Emails to TANF-CS Programs) to TANF and child support programs. The email will also include the study description and flier (Appendix B. Family Input Project Description and Pilot Testing Flier) as attachments. The email and flier will include a hyperlink to the expression of interest form (Instrument 1. Moving Forward Furthering Family Input Pilot: Expression of Interest form), which prospective study sites can fill out to express interest in participating in the study.

After a four- to six-week period for prospective study sites to submit an expression of interest, the study team will assess the number and quality of submissions. If the study receives more expressions of interest than the study can reasonably contact for a follow-up discussion, the study team will prioritize having follow-up discussions with prospective sites that appear to align most closely with the selection criteria outlined above. In those cases, the study team will send an email requesting to schedule a 30-minute telephone conversation (Appendix C. Family Input Example Direct Emails to TANF-CS Programs) to learn more about prospective study sites’ interests in participating (Instrument 2. Prospective site conversation guide). All prospective study sites will be notified of the study team’s decision regarding their participation (Appendix C. Family Input Example Direct Emails to TANF-CS Programs).

Types of interview respondents, data collection instruments to be used, and the approach for respondent selection are explained below.

Prospective site expression of interest web-based questionnaire (Instrument 1. Expression of interest form). The expression of interest form will be completed once by a manager-level program staff person. The study team assumes that no more than 24 programs will express interest. The expression of interest form will take approximately 10 minutes to complete.

Prospective site tele- or video-conference semi-structured interview guide (Instrument 2. Prospective site conversation guide). The study team will facilitate conversations with 16 programs that submitted an expression of interest form. The study team will choose the 16 programs for prospective site conversations based on the factors described in Section B2. The prospective site conversations will each last approximately 30 minutes.
Lead Implementation Team member web-based questionnaire (Instrument 3. Questionnaire about Family Input Resources for Implementation Team Lead). The questionnaire will be administered to the lead member of each of the eight pilot site’s Implementation Team, once per month during the 12-month pilot testing period and will take 15 minutes to complete.
Semi-structured interview guide for TANF and child support Implementation Team members (Instrument 4. Implementation Team interview guide). 60-minute semi-structured tele- or video conference interviews will be conducted with each site’s Implementation Team, around months six and twelve of the pilot period. The study aims to interview all members of each Implementation Team (up to eight staff) individually.
Focus groups with TANF and child support Implementation Team members (Instrument 5. Implementation Team focus group guide). The study will conduct a 60-minute video conference focus group with each site’s Implementation Team (up to eight staff) about four months after the pilot testing period ends.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

The data collection instruments were developed by content experts at Mathematica and MEF Associates, under contract to ACF, to capture essential data for the study’s guiding questions. The content was informed by the study’s prior literature scan, collaboration with the study’s Expert Workgroup (human services program administrators and people with lived experience in TANF and child support programs), and the design and content of the Family Input Resources toolkit. The instruments were closely examined to confirm that they are streamlined and will not collect duplicative data across respondents.

The study team does not intend to conduct a pre-test of the data collection instruments (see Section B7 for a description of testing survey programming). The expression of interest web-based questionnaire and prospective site semi-structured interview guide will be used for recruitment and site selection, and were developed to meet the study’s specific needs. The web-based questionnaire for Implementation Team Leads is intended to systematically document each site’s successes and challenges engaging families in program improvement, which materials in the Family Input Resources toolkit sites have used and how, and opinions and feedback about the materials. The semi-structured Implementation Team interview guide asks members about their experience using the Family Input Resources toolkit and engaging families in their program improvement effort. The Implementation Team focus group guide will help the study team understand how the full Implementation Team used the Family Input Resources toolkit and engaged families since the end of the pilot testing period.

Interviewers will ensure they use the program names, acronyms, and terminology that is appropriate for each respondent to understand the questions that are asked. Through ongoing technical assistance activities, interviewers will learn from each program’s Implementation Team lead how to word questions to take into account site-specific titles or terminology.

B4. Collection of Data and Quality Control

Expression of interest web-based questionnaire

Each prospective program site will fill out one expression of interest form, which is a questionnaire that will take approximately 10 minutes to complete. The web-based questionnaire will be administered via QuestionPro, an online survey tool appropriate for collecting high level standardized information. The expression of interest form will include an introduction with a brief description of the study and pilot opportunity.

Prospective site semi-structured interview guide

The study team will speak with 16 prospective pilot sites that completed the expression of interest form. The study team will e-mail the staff person who filled out the expression of interest form to schedule the 30-minute conversation. The study team expects these conversations will be conducted with one staff person from the program.

The interviewers will offer privacy assurances as part of the introduction to the conversations. A notetaker will take notes during the conversation.

Web-based questionnaire

Each Implementation Team lead will receive an email (Appendix D. Family Input Invitation Emails for Data Collection) inviting them to complete the questionnaire each month during the 12-month pilot testing period. The questionnaire will take about 15 minutes to complete.

The web-based questionnaire will be administered via QuestionPro. The web-based questionnaire will include an introduction explaining to respondents that (1) responses to the questionnaire will be used for research purposes, (2) information will not be attributed to questionnaire respondents, and (3) information about the respondents will not be made public. Respondents will be asked to submit completed questionnaires within one week of receipt.

Semi-structured interviews

Each Implementation Team member will be emailed (Appendix D. Family Input Invitation Emails for Data Collection) to schedule one-hour tele- or video-conference interviews (Instrument 4. Implementation Team interview guide). The interviews will take place in the sixth and twelfth months of the pilot testing period. Most interviews will be conducted on a one-on-one individual basis although some may be small group (i.e., no more than three individuals) if multiple staff were involved in similar roles within the program improvement effort.

The interviewers will offer privacy assurances as part of the introduction to the interview. With permission from respondents, interviews will be recorded to support the notes taken during the interviews. If respondents do not consent to be recorded, notes will be taken in lieu of a recording.

Post-piloting focus groups

About four months after the end of the 12-month pilot testing period, the study team will conduct virtual focus groups with each site’s full Implementation Team. The study team will send a scheduling email to each Implementation Team lead (Appendix D. Family Input Invitation Emails for Data Collection) to schedule one-hour focus groups (Instrument 5. Implementation Team focus group guide).

The facilitators will offer privacy assurances as part of the introduction to the focus group. The facilitator will ask focus group members to keep responses private within the group. With permission from respondents, focus groups will be recorded to support the notes taken during the conversations. If respondents do not consent to be recorded, notes will be taken in lieu of a recording.

Quality control

For the web-based questionnaire, all questionnaires will be reviewed for completeness. The web-based questionnaire will also include contact information for study team staff who can answer respondents’ questions.

Study team staff responsible for conducting semi-structured interviews and focus groups will participate in data collection training to ensure data are collected systematically and consistently. Topics to be discussed in the training include interviewing and focus group facilitation techniques, protecting privacy, and procedures for ensuring data security. The training will also include reviewing the data collection instruments to ensure full understanding of their purpose and the collection of comparable, complete, and high-quality data.

After the interview and focus group data are collected during pilot testing, data for each site will be entered into a standard template. Completed templates will be reviewed to check for gaps in information or inconsistencies. The study team will follow up with Implementation Teams directly to clarify any gaps in information or inconsistencies that may need to be addressed.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The study is not designed to produce representative or generalizable findings and site participation is wholly at the site’s discretion. Response rates will not be calculated or reported.

It is anticipated that in each site, Implementation Team members will participate in the interviews and focus groups and the team lead will complete the monthly questionnaire. To maximize response rates, within selected sites, staff interviews will be scheduled to accommodate respondents’ availability and scheduling needs. If there is an instance when a targeted respondent is unable to meet at the scheduled time, a member of the project team will reschedule the call at a more convenient time. For the monthly questionnaire, interviews, and focus groups, the study team will send tailored reminders if Implementation Team members are not responsive to data collection requests.

In total, it is expected that:

The expression of interest web-based questionnaire will receive no more than 24 responses from prospective sites.
The prospective site semi-structured interviews will be facilitated with no more than 16 prospective sites.
The web-based questionnaire will be administered to 8 Implementation Team leads on a monthly basis for a total of 12 times.
Semi-structured one-on-one or small group interviews will be conducted with all site Implementation Team members (up to 8 staff per site and up to 64 respondents total) twice.
Focus groups with all implementation site team members (up to 64 respondents split into 8 site-specific focus groups) will be held once.

Non-Response

The study is designed to produce accurate descriptions of pilot sites and does not need to secure a set number of sites or participation rate to be of value to the field. As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated.

Maximizing Response Rates

During recruitment and before inviting a site to participate in pilot testing, the study team will explain the expectations of participation. The study team is likely to maximize response rates by selecting sites that demonstrate a motivation and eagerness to participate. After sites are selected, the study team will emphasize the importance of participation in all data collection activities through the ongoing provision of technical assistance activities to the site. The scheduling of tele- and video conference interviews and focus groups will be flexible to accommodate the scheduling needs of respondents.

B6. Production of Estimates and Projections

The data collected is for internal study team use only and will be used for descriptive purposes. Data will not be used to make any population estimates, either for internal use or dissemination.

B7. Data Handling and Analysis

Data Handling

After the expression of interest web-based questionnaire and web-based questionnaire about the Family Input Resources are programmed into QuestionPro, several study team members who did not program the questionnaires will test the survey branching logic and response fields to ensure that the survey is programmed correctly and that only the expected types of responses may be entered. After the first administration of the questionnaires, study team members will review the data for quality and completeness and, if necessary, revise and retest the programming. Study team members will continue to review data quality after each administration of the questionnaires.

All interview and focus group questions are open-ended, and notes will be taken during both types of information collection activities. Interview and focus group audio files and notes will be stored in an encrypted project folder on Mathematica’s network.

Data Analysis

There are a few close-ended questions in the expression of interest web-based questionnaire and web-based questionnaire about the Family Input Resources. These quantitative data points will be used to provide descriptive statistics. The former to help assess the extent to which a prospective site knows where they want to focus their program improvement effort and how ready they feel to engage families. The latter to help assess the extent to which certain sections of the Family Input Resources toolkit are used by pilot sites, the extent to which the resources are useful, and the extent to which the Implementation Team lead would recommend the resources to someone else in the human services field.

Summary templates will be used to systematically summarize and organize key information collected during and after pilot testing. Each site will have a summary template that summarizes all of the information collected from the site. Data will be aggregated into a master file and analyzed in aggregate and by subgroups (for example, TANF versus child support programs).

The project team will use manual coding to analyze key topics or themes that arise during interviews, focus groups, and in the open-ended web-based questionnaire items. The study team will develop codes while reviewing the responses, identifying themes or patterns that can inform the study’s guiding questions.

Data Use

The study team will revise and update the Family Input Resources toolkit based on findings from this study. The final, updated version of the Family Input Resources toolkit will be made available to TANF, child support, and other humans services program administrators, staff, and the families they serve.

The findings from this study will be incorporated into reports and other technical assistance resources that will be shared publicly to provide timely information to TANF and child support programs about incorporating family input into program improvement activities. These documents will describe the data collection and analysis process and note that the results are not generalizable to improve understanding of how to properly interpret study findings.

B8. Contact Person(s)

Project leaders from the Office of Planning, Research, and Evaluation (OPRE), Mathematica, and MEF Associates who designed and will collect and analyze the data are as follows:

Lisa Zingman, Administration for Children and Families, OPRE, Federal Project Officer ([email protected])
Rebekah Selekman, Mathematica, Project Director ([email protected])
Jeanette Holdbrook, Mathematica, Deputy Project Director ([email protected])

Attachments

Instruments

Instrument 1. Expression of Interest form

Instrument 2. Prospective site conversation guide

Instrument 3. Questionnaire about Family Input Resources for Implementation Team Lead

Instrument 4. Implementation Team interview guide

Instrument 5. Implementation Team focus group guide

Appendices

Appendix A. Family Input OFA-OCSE and APHSA-NCCSD Support Email Examples

Appendix B. Family Input Project Description and Pilot Testing Flier

Appendix C. Family Input Example Direct Emails to TANF-CS Programs

Appendix D. Family Input Invitation Emails for Data Collection

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Mark Ezzo
File Modified	0000-00-00
File Created	2023-07-31

Supporting Statement B

Family Input_Formative ACF Program Support_0970-0531_SSB GenIC- OPRE Pilot_final_clean.docx

Formative Data Collections for ACF Program Support