Examining Equity of the Administration for Children and Families’ Healthy Marriage and Responsible Fatherhood (HMRF) Client Surveys
Administration for Children and Families Generic for Engagement Efforts
0970 – 0630
Supporting Statement Part A
Justification
August 2024
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
A1. Necessity for the Data Collection
The Office of Planning, Research, and Evaluation (OPRE), within the Administration for Children and Families (ACF), proposes to collect information to better understand aspects of equity for the Healthy Marriage and Responsible Fatherhood (HMRF) performance measures by learning about (1) clients’ opinions on survey questions that ACF uses to assess the performance of HMRF programs and (2) clients’ experiences taking the surveys.
This information collection request is to conduct focus groups and/or interviews to examine how HMRF clients from different populations and communities experience questions on seven existing HMRF client surveys (applicant characteristics [administered to all client populations] and population-specific entrance and exit surveys for three program types: HM for youth, HM for adults, and RF for community fathers; OMB Control Number 0970-0566). Under contract to ACF, the Mathematica study team will work with grant recipients to recruit clients to participate in the focus groups and/or interviews. The findings from the information collection will inform revisions to the performance measures for a potential cohort of grant recipients that could receive five-year grant awards in 2025. ACF does not intend for this information to be used as the principal basis for public policy decisions.
The Executive Order (EO), Advancing Racial Equity and Support for Underserved Communities Through the Federal Government (EO 13985)1 emphasizes consulting with communities that have been historically underserved by Federal policies and programs and those with lived experience in ACF programs. The EO on Further Advancing Racial Equity and Support for Underserved Communities Through the Federal Government followed in 2023 and built on EO 13985, calling upon agencies to increase engagement with underserved communities. To equitably and accurately measure HMRF clients’ characteristics and their program experiences, the data must reflect the strengths, needs, and experiences of multiple cultural groups and respectfully represent varying cultural values, beliefs, and judgments.2 By examining how different populations and communities experience the survey questions and data collection procedures, ACF can better use the measures to assess whether programs are equitably serving and benefitting their clients.
There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.
A2. Purpose of Survey and Data Collection Procedures
Overview of Purpose and Use
The primary purpose of this information collection is to examine whether the HMRF client surveys—which ACF uses to assess program performance—are aligned with a diverse range of cultural values and equitably assess HMRF programs’ potential influences on clients’ lives.
HMRF programs strive to strengthen families by improving their relationships, parenting, and economic stability. The Office of Family Assistance (OFA)—also within ACF—has a long-standing partnership with OPRE to support HMRF grant recipients with collecting and using data on their programs and clients. This information collection will use an equity lens to examine the questions in the client surveys (that is, the applicant characteristics, entrance, and exit surveys) and the data collection process for three types of HMRF grant recipients: (1) Healthy Marriage programs for youth, (2) Healthy Marriage programs for adults, and (3) Responsible Fatherhood programs for community fathers. ACF and HMRF grant recipients use these surveys to assess client characteristics and program outcomes.
The performance measures for the cohort of HMRF grant recipients that received five-year grant awards in 2020 drew on multiple sources. Many measures were the same as those used with the 2015 HMRF grant recipient cohort, which ACF selected based on the following criteria: (1) had been used successfully in prior studies with similar populations, whenever possible; (2) were appropriate for populations typically served by HMRF programs; (3) captured relevant short-term changes that might be expected by the end of the program; (4) were free and publicly available; (5) enabled clients to respond to questions themselves (no interviewer needed); and (6) had good statistical properties, including reliability and validity. In preparation for the 2020 HMRF grant recipient cohort, some measures were modified based on public comments, feedback from grant recipients and federal staff, and analyses of the performance measure data.
Although the measures were examined and revised for the 2020 HMRF grant recipient cohort, a recent study conducted as a part of OPRE’s Building Usage, Improvement, and Learning with Data in HMRF Programs (BUILD-HMRF) project offered an opportunity to reexamine them with a specific emphasis on equity. During this study, the BUILD HMRF team spoke with HMRF program staff, other practitioners, researchers, and a few HMRF clients. The team asked each question in the study of nine or fewer people. Staff from some grant recipients shared that their program clients expressed negative sentiments regarding some survey questions, indicating that the questions might not be universally appropriate or meaningful for all cultures. Respondents also provided suggestions for improving the measures, including the use of more strengths-based language in survey items. Examining the surveys through an equity lens enables ACF to address these concerns so the surveys are culturally sensitive and relevant to the diverse populations served by the programs. To hear a variety of perspectives, ACF seeks to hear from a broader range of HMRF clients so their voices and experiences can inform potential changes to the surveys.
To conduct this assessment, staff from Mathematica, the contractor carrying out BUILD-HMRF, will facilitate focus groups and/or interviews with HMRF clients who have completed the program workshops and have either completed the exit survey or declined to take the exit survey. Focus groups facilitate group interactions and exchanges that can uncover shared experiences and cultural patterns in responses. Facilitators of the focus groups will incorporate cognitive interviewing techniques, a common approach for identifying issues with survey questions, to prompt discussion and draw out clients’ nuanced reactions to the survey questions. The facilitators will use group activities, think-aloud interviewing, and verbal probing to encourage clients to describe their interpretations of a question, relevance or appropriateness of the questions, ease or difficulty in answering questions, and completeness or gaps in the response categories.3, 4 Focus groups facilitate group interactions and exchange that can uncover shared experiences and cultural patterns in responses. In situations where a focus group cannot be arranged, the Mathematica team will conduct a limited number of one-on-one cognitive interviews instead.
The intended uses of the information include informing the revisions to the HMRF surveys and data collection guidance for the potential 2025 grant recipient cohort. High-level feedback might be incorporated into a public report summarizing ACF’s proposed revisions to the surveys and data collection guidance. Information from the surveys is used to assess grant recipient program performance, inform program improvement, and advance learning in the broader HMRF field.
Making opportunities for ongoing, two-way collaborative and actionable communications between ACF and its state, local and/or Tribal partners, program participants, communities served or affected by ACF programs, and or others experienced with or interested in ACF programs or similar programs.
Informing program improvements
Informing program planning
Informing planning for new funding
Informing program implementation
Overview of Information Collections
Information Collection Title |
Respondent, Content, Purpose of Collection |
Mode and Duration |
Grant recipient recruitment protocol |
Respondents: Staff from grant recipients that have expressed interest in participating in the study.
Content: Questions about grant recipient’s workshops and the characteristics of clients who attend them.
Purpose: To provide information about whether and how to include the grant recipient in the study.
|
Mode: Phone
Duration: 30 minutes |
Demographic survey |
Respondents: Clients who have recently or will soon complete an HM or RF workshop.
Content: Questions about respondents’ demographic characteristics, such as race and ethnicity, gender identity, age and grade (for youth), and marital/relationship status (for adults).
Purpose: To provide information about the demographic characteristics of focus group or interview respondents.
|
Mode: Online or paper
Duration: 5 minutes |
HM youth focus group and interview protocols |
Respondents: Clients, ages 13 to 24, who have recently or will soon complete an HM youth workshop
Content: Discussion prompts about respondents’ understanding and reactions to the content, phrasing, and response categories for the HM program client survey questions and their experiences completing the surveys.
Purpose: To generate insights that will inform revisions to the HM program client surveys and performance measures for the expected 2025 cohort of grant recipients. |
Mode: Virtual focus group or interview
Duration: 60 minutes |
HM adult focus group and interview protocols |
Respondents: Clients, ages 18 and older, who have recently or will soon complete an HM adult workshop.
Content: Discussion prompts about respondents’ understanding and reactions to the content, phrasing, and response categories for the HM program client survey questions and their experiences completing the surveys.
Purpose: To generate insights that will inform revisions to the HM program client surveys and performance measures for the expected 2025 cohort of grant recipients. |
Mode: Virtual focus group or interview
Duration: 60 minutes |
RF adult focus group and interview protocols |
Respondents: Clients, ages 18 and older, who have recently or will soon complete an RF workshop.
Content: Discussion prompts about respondents’ understanding and reactions to the content, phrasing, and response categories for the RF program client survey questions and their experiences completing the surveys.
Purpose: To generate insights that will inform revisions to the RF program client surveys and performance measures for the expected 2025 cohort of grant recipients. |
Mode: Virtual focus group or interview
Duration: 60 minutes |
Processes for Information Collection
The Mathematica team will work with ACF to identify current HMRF grant recipients to contact. The goal is to reach out to grant recipients that serve one of the following client populations: (1) HM youth (ages 13 to 24); (2) HM adults (ages 18 or older), and (3) RF community fathers (men 18 and older with children who reside in the community). The study will prioritize grant recipients that serve a large proportion of clients from four groups: (1) African American (non-Hispanic), (2) American Indian or Alaska Native (non-Hispanic), (3) Asian American or Pacific Islander (non-Hispanic), and (4) Hispanic or Latino/a/x (any race). ACF will also consider contacting grant recipients that express interest in participating after learning about the study plans.
Once the grant recipients have been identified for potential participation, OFA Family Assistance Program Specialists (FPSs), who oversee the grants, will email grant recipients (Appendix A: Grant recipient outreach) asking them to participate in the data collection. They will send up to two follow-up emails (Appendix A: Grant recipient outreach) to grant recipients that do not respond. Mathematica staff will be included on these emails.
Mathematica staff will arrange a phone call to speak to grant recipients that agree to participate. During that phone call, Mathematica staff will describe the data collection and answer any questions the grant recipient staff have (Instrument 1: Grant recipient recruitment protocol).
Working with Mathematica staff, grant recipients will identify a current or planned program group workshop that will end within two to four months. Grant recipient staff will ask clients if they are interested in participating in the focus group. They will ask interested adult clients (including HM youth clients aged 18 and older) to complete a consent form. Staff from grant recipients that serve HM youth younger than 18 will ask parents or legal guardians to complete a consent form before data collection, and youth will complete an assent form. The consent and assent forms, which grant recipient staff can administer electronically or on paper, can be found in Appendix B: Participant consent.
After clients complete the consent and assent forms, grant recipient staff will ask them to complete a short questionnaire about their demographic information (Instrument 1: Participant demographics survey). The Mathematica team will use this information to understand the demographic composition of participating clients to provide context for the findings. In locations where the workshops occur in person, the grant recipient staff will send the completed consent forms, assent forms (when applicable), and questionnaires to Mathematica in a secured and tracked postal package. For programs that conduct workshops virtually, Mathematica will provide the consent and assent forms and the Participant Demographics survey in QuestionPro, or a similar secured web-based survey platform. Mathematica will provide the grant recipient staff with QR codes to share with clients so they can access required forms and the survey via the web.
Participating grant recipients will host an additional or extended workshop session after consenting clients complete the exit survey, as part of the standard process for performance measure data collection (OMB Control Number 0970-0566). The additional or extended workshop session could be virtual or in person; in either case, the focus group will be facilitated virtually by Mathematica staff. The grant recipient staff will link to the virtual focus group, using the information that Mathematica provides.
Mathematica staff will then conduct a 60-minute focus group (Instruments 3 – 5) with the clients using video conferencing software, such as WebEx or Zoom. During the focus group, Mathematica staff will ask for clients’ feedback on the experience of taking the surveys and their opinions on specific survey questions. The focus group will mainly emphasize questions from the exit survey that are also asked on the entrance survey, which will enable ACF and grant recipients to assess clients’ attitudes and behaviors when they begin and end the workshops. The focus group may also explore additional exit survey questions that address clients’ experiences during the program, and a few questions on the Applicant Characteristics survey (such as gender identity).
If a grant recipient is not able to arrange a focus group—for example, no program workshops will end in the specified time period—the Mathematica team may work with staff to schedule interviews with their clients (Instruments 3 – 5). The interviews will be conducted virtually with the client in a location of their choosing.
The focus groups and interviews will only include clients who had completed the HMRF exit survey or who completed the workshops but declined to complete the survey. Otherwise, clients’ answers to the exit survey questions would likely be affected by participation in the focus group or interview, which could distort program monitoring and the analysis that ACF and grant recipients conduct to measure clients’ change over time.
A3. Improved Information Technology to Reduce Burden
The Mathematica study team will conduct focus groups with youth and adults using a video conferencing software, such as Zoom. After obtaining permission from each participant, the Mathematica study team will video and audio-record all virtual focus groups and interviews. This is to capture information accurately without requiring clients to repeat themselves. The Mathematica study team will use the recordings to supplement the notes taken during each session. The video recordings will also allow for efficient transcript production, as draft transcripts can be automatically produced following the recording of these groups for review and cleaning by the Mathematica study team for accuracy.
The demographic survey and the consent and assent forms will be collected either by paper or electronically, based on the preference of each participating grant recipient. If collected electronically, Mathematica will use a data collection platform, QuestionPro for the collection of these data.
A4. Efforts to Identify Duplication
ACF is collecting information that is not available elsewhere.
A5. Involvement of Small Organizations
The potential exists for data collection activities to affect grant recipients that are small entities. Current data collection efforts are designed to minimize the burden on all organizations involved, including small businesses and entities, by collecting only critical information.
A6. Consequences of Less Frequent Data Collection
This is a one-time data collection.
A7. Special Circumstances
There are no special circumstances for the proposed data collection efforts.
A8. Federal Register Notice and Consultation
Federal Register Notice and Comments
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection request for a new umbrella generic clearance. The notice was published on December 11, 2023, (88 FR 85890), and provided a sixty-day period for public comment. ACF did not receive any comments on the first notice. A second notice was published, allowing a thirty-day period for public comment (89 FR 12352), in conjunction with submission of the request to OMB. ACF did not receive any comments on the second notice.
Study plans and protocols were reviewed by Dr. Lashawn Richburg-Hayes, Vice President for Education at Westat-Insight. Dr. Richburg-Hayes acts as an equity consultant for this data collection.
A9. Tokens of Appreciation for Respondents
It is extremely important to provide those with lived experience, experts, staff, and others providing their feedback for these efforts with equitable compensation or tokens of appreciation for participation. As noted in a 2022 report by the Office of the Assistant Secretary for Planning and Evaluation5 this “helps ensure a diverse population with varied views can participate.”
Consistent with OPRE’s umbrella clearance, ACF intends to offer a token of appreciation in the amount of $75 to each focus group or interview participant in recognition of their lived experience and expertise and to defray the incidental costs and barriers to participating, such as finding childcare, taking time off of work, transportation to a location with internet, or paying for technology costs (such as minutes or a data plan for an interview outside of the grant recipient’s location). The tokens of appreciation may mitigate financial and time-constraint barriers that socioeconomically disadvantaged families may face.6
As such, Mathematica plans to provide tokens of appreciation in the following amounts:
Instrument |
Time to Complete |
Token of Appreciation |
Participation in focus group or interview |
60 minutes |
$75 |
This proposal aligns with leading practices for ethical engagement of those with lived expertise and advancing equity for populations who have been historically underserved (as noted in section A1, advancing equity is a priority, as highlighted in the referenced EOs in that section). The proposed token of appreciation is intended to recognized the value of the information provided by clients, helps to remove barriers to participation, and affirms that the contributions from those with lived experience are as valuable as those from other experts.
A10. Privacy of Respondents
Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law.
The grant recipient will send Mathematica personally identifiable information (PII) for clients that participate in the focus group or interview so that Mathematica can directly send those clients the token of appreciation. Mathematica will not connect the PII to any information from the demographic questionnaires, focus groups, or interviews. Any PII transmitted from the grant recipient to Mathematica will be done via a secured, password protected file or through a secure upload to a restricted Box site on Mathematica’s secured server.
The Mathematica study team will destroy all electronically stored information as specified by the contractual requirements.
A11. Sensitive Questions
The focus group or interview questions may cover topics that some consider sensitive in nature, including opinions and experiences with relationships and marriage, parenting and co-parenting, economic stability, and personal development. These are the same sensitive topics covered in the applicant characteristics, entrance, and exit surveys that clients complete as part of program services (OMB Control Number 0970-0566). Before starting the focus group or interview, the Mathematica study team will inform clients that their participation is voluntary, that during the discussion they can decline to answer any of questions without consequence if responding would make them feel uncomfortable, and that they do not have to disclose any responses to items on the demographics survey or their responses on the client program surveys.
Consent and assent forms (Appendix B: Participant consent) will be completed as either hard copies or electronically, depending on the preference of each site and the mode in which they hold their program sessions.
An Institutional Review Board (IRB) will review all instruments and related materials, including the consent and assent forms.
A12. Estimation of Information Collection Burden
Burden Estimates
The Mathematica study team will reach out to grant recipients that express interest in participating in the study. In a 30-minute phone call, the study team will provide more information about participating and confirm information about the grant recipient’s workshops. The team expects two staff from the grant recipient to participate in each call. The team will use information from the call to identify workshops to include in the study or whether the team will conduct interviews with selected clients.
ACF expects that each focus group or interview, regardless of population or instrument, will take no longer than 60 minutes. Mathematica plans to conduct no more than 12 focus groups total, with up to 8 clients per group, not to exceed a total of 96 clients. As noted, if a grant recipient is not able to recruit sufficient clients for a focus group, Mathematica may conduct one-on-one interviews as needed.
Before
participating in a focus group or interview, clients will respond to
a short, 5-minute demographic questionnaire so the study team can
assess the demographic diversity of the final sample.
Cost Estimates
For cost calculations for grant recipient staff, we use the average wage for Social and Community Service Managers from the U.S. Bureau of Labor Statistics, Occupational Employment Statistics (OES), 2023.7 For cost calculations for clients, average hourly wage estimates have been based on the federal minimum wage of $7.25, also from the 2023 OES.8 The minimum wage estimates are appropriate for employment and salary levels for HMRF adult clients, who have relatively high proportions of unemployment and for youth program clients who may be working part-time minimum wage jobs.
For each instrument in the table below, ACF calculated the total annual costs by multiplying the annual burden hours by the average hourly wage. It is assumed that all 96 client respondents who complete the demographic survey will participate in a focus group or interview.
Instrument |
Total Number of Respondents |
Total Number of Responses Per Respondent |
Average Burden Hours Per Response |
Total Burden Hours |
Average Hourly Wage |
Total Annual Cost |
Instrument 1. Grant recipient recruitment protocol |
60 |
1 |
0.5 |
30 |
$40.10 |
$1,203 |
Instrument 2. Participant demographic survey |
96 |
1 |
0.08 |
8 |
$7.25 |
$58 |
Instrument 3. Healthy Marriage youth focus group and interview protocol |
32 |
1 |
1 |
32 |
$7.25 |
$232 |
Instrument 4. Healthy Marriage adult focus group and interview protocol |
32 |
1 |
1 |
32 |
$7.25 |
$232 |
Instrument 5. Responsible Fatherhood community father focus group and interview protocol |
32 |
1 |
1 |
32 |
$7.25 |
$232 |
Totals: |
156 |
|
|
134 |
|
$1957 |
A13. Cost Burden to Respondents or Record Keepers
There is no cost to respondents for participating in this research study, other than the time associated with participating in the focus group or interview.
There is a cost to the grant recipients that choose to participate in this study. Staff at each site will be asked to assist with recruiting clients for the focus group or interview, organize an additional workshop session to allow sufficient time for a group discussion, disseminate and collect consent (and if applicable youth assent) forms and a short demographic survey.
The participation of the grant recipients is vital to the study’s ability to directly engage with a diverse group of communities and individuals. Directly engaging the communities ACF serves and including these individuals in ACF research is in line with the following priorities of the current Administration and HHS:
Advancing Racial Equity and Support for Underserved Communities Through the Federal Government (EO 13985)
Further Advancing Racial Equity and Support for Underserved Communities Through the Federal Government
Presidential Memorandum on Restoring Trust in Government through Scientific Integrity and Evidence-Based Policy Making
ACF Evaluation Policy
HHS Strategic Plan FY 2022-2026,
ASPE’s Methods and Emerging Strategies to Engage People with Lived Experience (2021)
ASPE’s Recruiting Individuals with Lived Experience (2022)
Consistent with the guidance documents referenced, and to ensure the grant recipients are able to facilitate engagement with a variety of people with diverse experiences and perspectives, we propose to provide a $1,500 stipend to each participating grant recipient that works with the team to arrange a focus group or interviews. The payment is to defray the costs of grant recipient staff taking time away from their regular responsibilities to assist in this data collection, such as collecting consent and arranging an additional or extended workshop session for the focus group. Although programs receive federal funding as grant recipients, the activities described in this information collection are beyond the requirements of the grants and participation in this study is not included in their grant budget.
A14. Estimate of Cost to the Federal Government
The total cost for the data collection activities under this current request will be $351,169.
A15. Change in Burden
This is for an individual information collection under the umbrella generic clearance for ACF engagement activities (0970-0630).
A16. Plan and Time Schedule for Information Collection, Tabulation and Publication
The following is the anticipated timeline for the data collection, pending OMB approval:
August 2024 – November 2024: Engage with HMRF clients
October 2024 – December 2024: Summarize findings
January 2025 – March 2025: Revise measures based on ACF feedback
March 2025 – May 2025: Develop and submit full OMB package with revised measures for a potential cohort of grant recipients that could receive five-year grant awards in 2025
A17. Reasons Not to Display OMB Expiration Date
All instruments will display the expiration date for OMB approval.
A18. Exceptions to Certification for Paperwork Reduction Act Submissions
No exceptions are necessary for this information collection.
Attachments
Instruments
Instrument 1: Grant recipient recruitment protocol
Instrument 2: Client demographic survey
Instrument 3: HM youth focus group and interview protocol
Instrument 4: HM adult focus group and interview protocol
Instrument 5: RF community fathers focus group and interview protocol
Appendices
Appendix A: Outreach to grant recipients
Appendix B: Client consent
1 Exec. Order No. 13985, 86 Fed. Reg. 7009 (January 20, 2021) and Exec. Order No. 14091, 88 Fed. Reg. 10825 (February 16, 2023).
2 Modified from the National Center for Cultural Competence. (n.d.) “Cultural Awareness.” Georgetown University, Center for Child and Human Development.
3 Beatty, P.C., and G.B. Willis. "Research Synthesis: The Practice of Cognitive Interviewing." Public Opinion Quarterly, vol. 71, no. 2, 2007, pp. 287-311.
4 Willis, G. Analysis of the Cognitive Interview in Questionnaire Design: Understanding Qualitative Research. Oxford University Press, 2015.
6 Hinojosa, M.S., Kadivar, H., Fernandez-Baca, D., Chisholm, T., Thompson, L.A., Stanford, J., and E. Shenkman. “Recruiting Low Income and Racially/Ethnically Diverse Adolescents for Focus Groups.” Maternal and Child Health Journal, vol. 18, 2014, pp. 1912-1918.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | OPRE OMB Clearance Manual |
| Author | DHHS |
| File Modified | 0000-00-00 |
| File Created | 2025-02-17 |