Instrument 3: Impact Evaluation Final Report Table Shells

Impact Evaluation Final Report Table Shells

Healthy Marriage and Responsible Fatherhood Grant Recipients

December 2024

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this collection is ##-#; this number is valid through [DATE]. Public reporting burden for this collection of information is estimated to average 10 hours, including the time for reviewing instructions, gathering and maintaining the data needed, reviewing the collection of information, and revising it. This collection of information is voluntary for individuals, but the information is required from HMRF grant recipients to retain a benefit (Authority: 42 U.S.C. 603[a][2]).

TIP: In all tables and figures in this template, the text in italics gives examples of the kind of information you would enter in these cells. Please use that italicized text as a guide and remove it before entering your own information. Please use a regular (not italic) font in your final tables. Instructions for completing all tables are in the instructions for the final impact report (in a separate file). Text in brackets indicates that you should customize the text to make sense for your study. Please renumber tables in your final report as necessary.

Table Il.1. Description of intended intervention, counterfactual components, and focal populations

Component	Curriculum and content	Dosage and schedule	Delivery	Focal population
Intervention
Relationship skills workshops	Healthy relationships curriculum: Understanding partner’s perspectives; avoiding destructive conflict; and communicating effectively	Twenty hours, with two-hour sessions twice a week or four-hour sessions every Saturday	Group lessons provided at the intervention’s facilities by two trained facilitators in every session	Married couples with low incomes
Parenting workshops	Parenting skills workshops: learning about the importance of a father-child relationship; creating a positive learning environment; and practicing communication skills	Thirty hours, with a one-hour session occurring three times a week	Group lessons provided at the intervention’s facilities by one trained facilitator in every session	Fathers who have a child younger than 24
Economic stability workshops	Resume preparation; interview and communication skills; appropriate work attire; financial literacy	Monthly two-hour workshops	Workshops are provided by one facilitator	Individual members of the couple who need help with a job search
Counterfactual
Economic stability workshops	Resume preparation; interview and communication skills; appropriate work attire; financial literacy	Monthly two-hour workshops	Workshops are provided by one facilitator	Individual members of the couple who need job search assistance

Notes: [Anything important to note about the information above.]

Table II.2. Staff characteristics, education, training, and development to support intervention and counterfactual components

Component	Staff characteristics, education, and initial training	Ongoing staff training
Intervention
Relationship skills workshops	Facilitators are male and female and hold at least a bachelor’s degree and received four days of initial training.	Facilitators receive a half-day of semiannual refresher training in the intervention’s curricula from study staff.
Parenting workshop	Facilitators are male and female and hold at least a bachelor’s degree and received four days of initial training.	Facilitators receive a half-day of semiannual refresher training in the intervention’s curricula from study staff.
Economic stability workshops	Facilitators are male and female and hold at least a bachelor’s degree and received two days of initial training.	Facilitators receive a half-day of semiannual refresher training in the intervention’s curricula from study staff.
Counterfactual
Economic stability workshops	Facilitators are male and female and hold at least a bachelor’s degree and received two days of initial training.	Facilitators receive a half-day of semiannual refresher training in the intervention’s curricula from study staff.

Notes: [Anything important to note about the information above]

Table III.1. Outcome measures used to answer primary research questions of the impact analysis

Research question #	Outcome name	Description of the outcome measure and its properties	Source of the measure	Timing of measure
	Level of affection	The outcome measure is a scale ranging from 1 to 5, with the outcome calculated as a sum of both partners’ responses to five survey items with values 1 (strongly disagree to 0 (strongly agree) measuring: Support (I feel supported by my partner) Intimacy (I feel close to my partner) Commitment (I think my partner is committed to me) Trust (I trust my partner) Friendship (I am friends with my partner) Cronbach’s alpha (if applicable): [enter number]	Local follow-up survey	Six months after intervention ends

Notes: [Anything to note about the information above]

Table III.2. Outcome measures used to answer secondary research questions for the impact analysis [italicized text is an example of how to fill the cells in]

Research question #	Outcome name	Description of outcome measure and its properties	Source of the measure	Timing of measure
	Relationship skills	The outcome measure is calculated as the sum of six items with values 1 (Yes) and 0 (No): I feel good about my ability to make a romantic relationship last I am very confident when I think of having a stable, long-term relationship I have the skills needed for a lasting, stable romantic relationship I accept my partners point of view even if I don’t agree with it I can recognize early on the warning signs of a bad relationship I know what to do when I recognize the warning signs of a bad relationship Cronbach’s alpha (if applicable): (enter number)	nFORM exit survey	At post-test (immediately after intervention ends)
	Parenting attitudes about relationship with child	The outcome measure is a scale ranging from 7–35, with the outcome calculated as the sum of seven items with values 1 (Never) to 5 (Always): How often do you feel disappointed with [Child]? How often do you wish that [Child] was different? How often do you feel proud of [Child]? How often do you feel angry or irritated with [Child]? How often do you accept [Child] the way he or she is? How often do you feel you and your child understand each other? How often do you and your child argue and fight? Cronbach’s alpha (if applicable): [enter number]	nFORM entrance survey	At pre-test (before intervention begins)

Notes: [Anything important to note about the information above]

Table III.3. Measures used to address implementation research questions [italicized text is an example of how to fill the cells in]

Implementation element	Research question	Measures
Fidelity	Were all intended intervention components offered and for the expected duration?	Total number of sessions delivered Average session duration, calculated as the average of the recorded session lengths (in minutes)
Fidelity	What content did the clients receive?	Total number of topics covered, calculated as the average of the total number of topics checked by each intervention facilitator in the daily fidelity tracking log or protocol Number of HMRF topics covered by other providers during the evaluation period, based on survey data
Fidelity	Who delivered services to clients?	Number and type of staff delivering services to study participants, such as the number of session facilitators and couples’ therapists Percentage of staff trained, calculated as the number of staff who were trained divided by the total number of staff who delivered the intervention
Fidelity	What were the unplanned adaptations to key intervention components?	List of unplanned adaptations, such as a change in setting, sessions added or deleted, and components cut
Dosage	How often did clients participate in the intervention on average?	Average number (or percentage) of sessions clients attended Percentage of the sample attending the required or recommended proportion of sessions Percentage of the sample that did not attend any sessions Participation in services similar to those offered by the HMRF program but from other sources, and number of hours received, based on survey data
Quality	What was the quality of staff–participant interactions?	Percentage of sessions with high-quality interactions, calculated as the percentage of observed interactions that study staff scored as “high quality”
Engagement	How engaged were clients in the intervention?	Percentage of sessions with moderate participant engagement, calculated as the percentage of sessions in which study staff scored participants’ engagement as “moderately engaged” or higher Average engagement rating, calculated as the average of engagement scale scores (ranging from 1–5, for example) across satisfaction surveys Reports of level of engagement in the intervention or in similar HMRF services, based on survey data
Context	What other HMRF programming was available to study participants?	Percentage of the sample receiving HMRF programming from other providers, constructed from clients’ survey data on experiences outside of the current intervention List of HMRF programming available to study participants outside of the current intervention, as described on the websites of other agencies in the community
Context	What external events affected implementation?	Percentage and total number of anticipated study participants not enrolled due to community issues, if any Number of sites or schools that were closed as a result of weather events or policy changes (unrelated to the HMRF programming), if any

Note: We used the word “clients” in this table for simplicity’s sake.

Table IV.1a. Individual sample sizes, by intervention status [Only use for studies with individual-level assignment; if your design uses cluster-level assignment, skip this table and use Table IV.1b instead]

Number of individuals	Intervention sample size	Comparison sample size	Total sample size	Total response rate	Intervention response rate	Comparison response rate
Assigned to condition	1a	1b	1c (= 1a + 1b)	n.a.	n.a.	n.a.
Contributed a baseline survey	2a	2b	2c (= 2a + 2b)	= 2c/1c	= 2a/1a	= 2b/1b
Contributed to [first follow-up survey (timing)]	3a	3b	3c (= 3a + 3b)	= 3c/1c	= 3a/1a	= 3b/1b
Contributed to [first follow-up (timing) outcomes (accounts for item nonresponse and any other analysis restrictions)]
Outcome 1	4a	4b	4c ( = 4a + 4b)	= 4c/1c	= 4a/1a	= 4b/1b
Outcome 2	5a	5b	5c ( = 5a + 5b)	= 5c/1c	= 5a/1a	= 5b/1b
Contributed to [second follow-up survey (timing)]	6a	6b	6c (= 6a + 6b)	= 6c/1c	= 6a/1a	= 6b/1b
Contributed to [second follow-up (timing) outcomes (accounts for item nonresponse and any other analysis restrictions)]
Outcome 1	7a	7b	7c ( = 7a + 7b)	= 7c/1c	= 7a/1a	= 7b/1b
Outcome 2	8a	8b	8c (= 8a + 8b)	= 8c/1c	= 8a/1a	= 8b/1b

Notes: [Anything important to note about the information above]

n.a. = not applicable

[TIP: For rows that account for item nonresponse and other analysis restriction, note that you may have different sample sizes for two outcomes of interest because of different rates of missing data for the outcomes. Please add a row for each outcome in each time period, as needed, to Indicate the sample sizes of those who contributed data for that outcome at that follow-up, accounting for item nonresponse and any other analysis restrictions. For example, for the first follow-up, if you have two primary outcomes (such as Outcome 1 and Outcome 2), you should include two rows for “Contributed to first follow-up (accounts for item nonresponse and other analysis restrictions),” one for the analysis sample for Outcome 1 and one for the analysis sample for Outcome 2.]

Table IV.1b. Cluster and individual sample sizes by intervention status [Only use for studies with cluster-level assignment; if your design uses individual-level assignment, skip this table and use Table IV.1a instead]

Number of:	Intervention sample size	Comparison sample size	Total sample size	Total response rate	Intervention response rate	Comparison response rate
Clusters
Clusters: At beginning of study	1a	1b	1c (= 1a + 1b)	n.a.	n.a.	n.a.
Clusters: Contributed at least one individual at baseline	2a	2b	2c (= 2a + 2b)	= 2c/1c	= 2a/1a	= 2b/1b
Clusters: Contributed at least one individual at first follow-up (timing)	3a	3b	3c (= 3a + 3b)	= 3c/1c	= 3a/1a	= 3b/1b
Clusters: Contributed at least one individual at second follow-up (timing)	4a	4b	4c (= 4a + 4b)	= 4c/1c	= 4a/1a	= 4b/1b
Individuals in non-attriting clustersa
Individual: At time that clusters were assigned to condition	5a	5b	5c (= 5a + 5b)	n.a.	n.a.	n.a.
Individual: Who consented	6a	6b	6c (= 6a + 6b)	= 6c/5c	= 6a/5a	= 6b/5b
Individual: Contributed a baseline survey	7a	7b	7c (= 7a + 7b)	= 7c/5c	= 8a/5a	= 8b/5b
Individual: Contributed to first follow-up survey (timing)	8a	8b	8c (= 8a + 8b)	= 8c/5c	= 9a/5a	= 9b/5b
Individual: Contributed to the impact analysis of outcome at first follow-up (timing), accounting for item nonresponse and any other analysis restrictionsb
Outcome 1	9a	9b	9c (= 9a + 9b)	= 9c/5c	= 9a/5a	= 9b/5b
Outcome 2	10a	10b	10c (= 10a + 10b)	= 10c/5c	= 10a/5a	= 10b/5b
Individual: Contributed to second follow-up survey (timing)	11a	11b	11c (= 11a + 11b)	= 11/5c	=11a/5a	=11b/5b
Individual: Contributed to the impact analysis of outcome at second follow-up (timing), (accounting for item nonresponse and any other analysis restrictionsb
Outcome 1	12a	12b	12c (= 12a + 12b)	= 12/5c	= 12a/5a	= 12b/5b
Outcome 2	13a	13b	13c (= 13a + 13b)	= 13/5c	= 13a/5a	= 13b/5b

^{a [}For all rows in this section, do not include individuals from clusters that dropped (attrited) over the course of the study. For example, if you randomly assigned 10 clusters (5 to each condition), and one intervention group cluster (e.g. school) dropped from the study, you would only include individuals in this section from the 9 clusters that did not drop from the study. Because the cluster-level response rate in the above rows already captures that dropped cluster, you do not need to count individuals from the lost clusters in your individual-level response rates.]

^b [See guidance in Section IV.A for defining your analytic sample(s).]

[TIP: For rows that account for item nonresponse and other analysis restriction, note that you may have different sample sizes for two outcomes of interest because of different rates of missing data for the outcomes. Please add a row for each outcome in each time period, as needed, to Indicate the sample sizes of those who contributed data for that outcome at that follow-up, accounting for item nonresponse and any other analysis restrictions. For example, for the first follow-up, if you have two primary outcomes (such as Outcome 1 and Outcome 2), you should include two rows for “Contributed to first follow-up (accounts for item nonresponse and other analysis restrictions),” one for the analysis sample for Outcome 1 and one for the analysis sample for Outcome 2.]

Notes: n.a. = not applicable.

Table IV.2 Summary statistics of key baseline measures and baseline equivalence across study groups, for individuals/couples completing the [follow-up timing] survey

Baseline measure	Intervention mean	Intervention standard deviation	Comparison mean	Comparison standard deviation	Intervention and comparison difference in means	p-value of test of difference in means	Effect size [strongly recommended]
Demographic characteristic 1
Demographic characteristic 2
Demographic characteristic 3
Demographic characteristic 4
Baseline measure of outcome 1
Baseline measure of outcome 2
Sample size		n.a.		n.a.	n.a.	n.a.	n.a.

Notes: [Effect sizes are calculated using (Hedges’ g or Cox’s index) formula.] or [ p-values are included in parentheses.]

n.a. = not applicable.

[Anything else important to note about the information above]

[TIP: Please present a baseline equivalence table for the sample of survey respondents at each follow-up.]

Table IV.3. Covariates included in the impact analyses

Covariate	Description of the covariate
Age	Age (in years) as of the baseline data collection
Baseline marital status	Marital status (1 = married; 0 = not married) as of the baseline data collection
Covariate 3	Description of covariate 3
Covariate 4	Description of covariate 4

Notes: [Anything to note about the analysis.]

Table V.1a. Post-intervention estimated effects using data from [survey follow-up time period] to address the primary research questions

Outcome measure	Intervention mean or %	Intervention standard deviation	Comparison mean or %	Comparison standard deviation	Intervention and comparison difference in means	p-value of test of difference in means	Effect size [strongly recommended]
Outcome 1
Outcome 2
Sample size		n.a.		n.a.	n.a.	n.a.	n.a.

Source: [Name for the Data Collection, Date. For instance, first follow-up surveys administered 12 months after the program.]

Notes: Effect sizes are calculated using (Hedges’ g or Cox’s index) formula. [Add here anything else to note about the analysis]. See Table III.1 for a more detailed description of each measure and section IV.C in Chapter IV for a description of the impact estimation approach.

**/*/+ Differences are statistically significant at the 0.01/0.05/0.10 levels, respectively.

n.a. = not applicable.

Table V.1b. Post-intervention tests of equivalent effects using data from [survey follow-up time period] to address the primary research questions

Outcome measure	Intervention mean or %	Intervention SD	Comparison mean or %	Comparison SD	Smallest effect size of interest in SD units	Equivalence interval	p-value of test of difference in means lower or equal to lower bound	p-value of test of difference in means greater or equal to upper bound	Equivalent effects established (Yes/No)
Relationship commitment scale (range 1 to 10)	9.5	2.0	9.4	2.1	0.05 SD	(-0.25, 0.25)	0.117	0.305	No
Attitudes toward relationship with child
Parenting attitudes
Job skills
Analytic sample size for outcome measure
Relationship commitment scale	95	n.a.	100	n.a.	n.a.	n.a.	n.a.	n.a.	n.a.
Attitudes toward relationship with child		n.a.		n.a.	n.a.	n.a.	n.a.	n.a.	n.a.
Parenting attitudes		n.a.		n.a.	n.a.	n.a.	n.a.	n.a.	n.a.
Job skills		n.a.		n.a.	n.a.	n.a.	n.a.	n.a.	n.a.

Source: [Name for the Data Collection, Date. For instance, first follow-up surveys administered 12 months after the program.]

Notes: [Add here any relevant information about how the tests of equivalent effects were conducted. For example, please indicate whether the intervention and comparison means are unadjusted or covariate-adjusted based on the specification of the final impact model. If covariate-adjusted means are used, then these should be used for equivalence testing]. The intervention condition in this evaluation refers to in-person delivery of the program, and the comparison condition refers to live-streaming delivery of the program. See Table III.1 for a detailed description of each measure and Section IV.C in Chapter IV for a description of the impact estimation approach.

n.a. = not applicable; SD = standard deviation.

* Differences are statistically significant at the 0.05 level.

Table V.1c. Post-intervention tests of equivalent effects using data from [survey follow-up time period] to address the primary research questions

Outcome measure	Intervention mean or %	Intervention SD	Comparison mean or %	Comparison SD	Smallest effect size of interest in SD units	Equivalence interval	90 percent confidence interval of difference in means	Equivalent effects established (Yes/No)
Relationship commitment scale (range 1 to 10)	9.5	s	9.4	2.1	0.05 SD	(-0.25, 0.25)	(–0.39, 0.59)	No
Attitudes toward relationship with child
Analytic sample size for outcome measure
Relationship commitment scale	95	n.a.	100	n.a.	n.a.	n.a.	n.a.	n.a.
Attitudes toward relationship with child		n.a.		n.a.	n.a.	n.a.	n.a.	n.a.
		n.a.		n.a.	n.a.	n.a.	n.a.	n.a.
		n.a.		n.a.	n.a.	n.a.	n.a.	n.a.

Source: [Name for the data collection, Date. For instance, first follow-up surveys administered 12 months after the program.]

Notes: [Add here any relevant information about how the tests of equivalent effects were conducted. For example, please indicate whether the intervention and comparison means are unadjusted or covariate-adjusted based on the specification of the final impact model. If covariate-adjusted means are used, then these should be used for equivalence testing]. The intervention condition in this evaluation refers to in-person delivery of the program, and the comparison condition refers to live-streaming delivery of the program. See Table III.1 for a detailed description of each measure and Section IV.C in Chapter IV for a description of the impact estimation approach. SD = Standard deviation; n.a. = not applicable.

Table V.2. Differences in means between intervention and comparison groups estimated using alternative methods (sensitivity analyses)

Outcome	Primary approach	No covariate adjustment	Name of sensitivity approach 2	Name of sensitivity approach 3
Primary research questions
Outcome 1
Outcome 2
Outcome 3

Source: [Name for the data collection, Date. For instance, Follow-up surveys administered six to eight months after the program.]

Notes: [Anything to note about the analysis]

**/*/+ Differences are statistically significant at the 0.01/0.05/0.10 levels, respectively.

Table V.3 Post-intervention estimated effects using data from [survey follow-up time period] to address the secondary research questions

Outcome measure	Intervention mean or %	Intervention standard deviation	Comparison mean or %	Comparison standard deviation	Intervention and comparison difference in means	p-value of test of difference in means	Effect size [strongly recommended]
Outcome 1
Outcome 2
Sample size		n.a.		n.a.	n.a.	n.a.	n.a.

Source: [Name for the data collection, Date. For instance, first follow-up surveys administered 12 months after the program.]

Notes: Effect sizes are calculated using (Hedges’ g or Cox’s index) formula. [Add here anything else to note about the analysis]. See Table III.1 for a detailed description of each measure and Section IV.F in Chapter IV for a description of the impact estimation approach. n.a. = Not applicable.

**/*/+ Differences are statistically significant at the 0.01/0.05/0.10 levels, respectively.

Appendix Tables and Figures to Supplement Final Impact Report

Appendix A: Logic Model (if applicable)

Logic model for [name of the intervention].

Paste the logic model for the intervention here.

Appendix B: Data and Study Sample

Table B.1. Key features of data collection for the impact analysis

Study group	Data source	Timing of data collection	Mode of data collection	Parties responsible for data collection	Start and end date of data collection
Intervention	nFORM entrance and exit surveys	Enrollment (baseline) End of intervention (eight months after enrollment)	In-person online survey	Program staff	September 2016 through January 2020
	Local evaluation survey	Three months after the end of the intervention (11 months after enrollment) Six months after the end of the intervention (14 months after enrollment)	Telephone survey	Evaluation staff	August 2017 through March 2021
Counterfactual	nFORM entrance survey	Enrollment (baseline)	In-person online survey	Program staff	September 2016 through January 2020
	Local evaluation survey	Eight-month follow-up 11-month follow-up 14-month follow-up	Telephone survey	Evaluation staff	August 2017 through March 2021

Table B.2. Key features of data collection for the implementation analysis

Implementation element	Research question	Data source	Timing and frequency of data collection	Party responsible for data collection
Fidelity	Were all intended intervention components offered and for the expected duration?	Workshop sessions in nFORM	All sessions delivered	Intervention staff
Fidelity	What content did the clients receive?	Fidelity tracking log or protocol; attendance logs; session observations	Every session for fidelity tracking and attendance logs; twice a year for session observations	Intervention staff for fidelity tracking and attendance logs; study staff for session observations
Fidelity	Who delivered services to clients?	Staff applications; hiring records; training logs	One time X months after start of implementation; annually	Intervention staff
Fidelity	What were the unplanned adaptations to key intervention components?	Adaptation request; work plan; six-month progress report; annual progress report	Annually; ad hoc	Intervention staff; study staff
Dosage	How often did clients participate in the intervention on average?	Workshop sessions and individual service contacts in nFORM; attendance logs	All sessions delivered	Intervention staff
Quality	What was the quality of staff–participant interactions?	Observations of interaction quality, using protocol developed by study staff	X percentage of sessions selected at random for observation	Study staff
Engagement	How engaged were clients in the intervention?	Observations of engagement, possibly using an engagement assessment tool; ratings from facilitator fidelity logs; engagement ratings from participant satisfaction surveys	Y percentage of sessions selected at random for observation	Study staff
Context	What other HMRF programming was available to study participants?	Interviews with staff from partnering agencies in the community; survey items on baseline and follow-up assessments; websites of other agencies in the community providing HMRF programming	Once a year; ad hoc	Study staff
Context	What external events affected implementation?	Interviews with community or county representatives; list of site or school closures	Once a year; ad hoc	Study staff

Note:

[TIP: The examples in the table use “clients” to avoid redundancy.]

CONSORT diagram

Instructions. Paste updated CONSORT diagram from your analysis plan here.

Appendix C: Baseline Equivalence

C.1. Baseline equivalence assessment

Instructions. Use Table C.1 to summarize baseline equivalence for the analytic sample of each outcome measure you used to estimate impacts (to answer the primary research questions). It is good practice to establish baseline equivalence for low-attrition RCTs. If the evaluation is an RCT with high attrition, an RCT with any other issue that compromised the random assignment design, or a QED, demonstrating baseline equivalence for the analytic sample of each outcome is required.

Table C.1. Summary statistics of key baseline measures and baseline equivalence across study groups, for individuals/couples completing [outcome measure #] at the [follow-up timing] follow-up

Baseline measure	Intervention mean	Intervention standard deviation	Comparison mean	Comparison standard deviation	Intervention and comparison difference in means	p-value of test of difference in means	Effect size [strongly recommended]
Demographic characteristic 1
Demographic characteristic 2
Demographic characteristic 3
Demographic characteristic 4
Baseline measure of outcome #
Sample size		n.a.		n.a.	n.a.	n.a.	n.a.

Notes: Effect sizes are calculated using (Hedges’ g or Cox’s index) formula. [Anything else important to note about the information above].

n.a. = Not applicable.

**/*/+ Differences are statistically significant at the 0.01/0.05/0.10 levels, respectively.

[TIP: Please present one baseline table for the analytic sample of each outcome on which impacts are estimated to answer the primary research questions.]

C.2. Statistical approach to constructing equivalent groups

Instructions. If the evaluation’s design is a QED, or it was originally an RCT but you had to construct equivalent groups using a statistical approach (so the design effectively became a QED) due to high attrition, lack of baseline equivalence, and/or another issue that compromised the random assignment, please describe the details (with text) about the statistical approach in this appendix.