Instrument 2 - FaMLE Impact Report Table Shells

Formative Data Collections for ACF Research

Instrument 2 - FaMLE Impact Report Table Shells

OMB: 0970-0356

Document [docx]
Download: docx | pdf







Final Impact Evaluation Report Tables Template for Healthy Marriage and Responsible Fatherhood Grantees

Shape1

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this collection is 0970-0356; this number is valid through 6/30/2021.  Public reporting burden for this collection of information is estimated to average 10 hours, including the time for reviewing instructions, gathering and maintaining the data needed, reviewing the collection of information, and revising it. This collection of information is voluntary for individuals, but the information is required from Grantees.




NOTE: In all tables and figures in this template, example information is included in italics. Please use as a guide and remove before completing tables/figures. Please use a regular font in your final tables. Instructions for completing all tables are included in the impact report instructions document.



Table Il.1. Description of intended intervention and counterfactual components and target populations

Component

Curriculum and content

Dosage and schedule

Delivery

Target Population

Intervention

Relationship skills workshops

Healthy relationships curriculum: Understanding partner’s perspectives; avoiding destructive conflict; and communicating effectively

20 hours, with 2-hour sessions occurring twice a week, or 4-hour sessions occurring every Saturday

Group lessons provided at the intervention’s facilities by two trained facilitators in every session

Low-income married couples

Economic stability workshops

Resume preparation; interview and communication skills; appropriate work attire; financial literacy

Monthly 2-hour workshops

Workshops are provided by one facilitator

Individual members of the couple who need job search assistance

Counterfactual

Economic stability workshops

Resume preparation; interview and communication skills; appropriate work attire; financial literacy

Monthly 2-hour workshops

Workshops are provided by one facilitator

Individual members of the couple who need job search assistance

Notes: [Anything important to note about the information above]

























Table II.2. Staff training and development to support intervention and counterfactual components

Component

Education and initial training of staff

Ongoing training of staff

Intervention

Relationship skills workshops

Facilitators are male and female and hold at least a bachelor’s degree and received four days of initial training.

Facilitators receive a half-day of semi-annual refresher training in the intervention’s curricula from study staff.

Economic stability workshops

Facilitators are male and female and hold at least a bachelor’s degree and received two days of initial training.

Facilitators receive a half-day of semi-annual refresher training in the intervention’s curricula from study staff.

Counterfactual

Economic stability workshops

Facilitators are male and female and hold at least a bachelor’s degree and received two days of initial training.

Facilitators receive a half-day of semi-annual refresher training in the intervention’s curricula from study staff.

Notes: [Anything important to note about the information above]



























Table IV.1a. Individual sample sizes by intervention status (Only use for studies with individual-level assignment; if your design uses cluster-level assignment, skip this table and use Table IV.1b instead)

Number of individuals

Intervention sample size

Comparison sample size

Total sample size

Total response rate

Intervention response rate

Comparison response rate

Assigned to condition

1a

1b

1c (=1a + 1b)

N/A

NA

N/A

Contributed a baseline survey

2a

2b

2c (=2a + 2b)

=2c/1c

=2a/1a

=2b/1b

Contributed to first follow-up survey (timing)

3a

3b

3c (=3a + 3b)

=3c/1c

=3a/1a

=3b/1b

Contributed to first follow-up (timing)

(accounts for item non-response and any other analysis restrictions)

4a

4b

4c (=4a + 4b)

=4c/1c

=4a/1a

=4b/1b

Contributed to second follow-up survey (timing)

5a

5b

5c (=5a + 5b)

=5c/1c

=5a/1a

=5b/1b

Contributed to second follow-up (timing)

(accounts for item non-response and any other analysis restrictions)

6a

6b

6c (=6a + 6b)

=6c/1c

=6a/1a

=6b/1b

Notes: [Anything important to note about the information above]





Reminder from instructions: For rows that account for item non-response and other analysis restriction, note that you may have very different sample sizes for two outcomes of interest because of very different rates of missing data for the outcomes. If this is the case, please add a row for each outcome in each time period, as needed. Indicate in a table note to which outcome the sample sizes apply. For example, if you have two primary outcomes and there was very different response rates on the items needed to construct these outcomes, you should include two rows for “Contributed to first follow-up (accounts for item non-response and other analysis restrictions)”: one for the analysis sample for outcome one and one for the analysis sample for outcome two.





























Table IV.1b. Cluster and individual sample sizes by intervention status (Only use studies with cluster-level assignment; if your design uses individual-level assignment, skip this table and use Table IV.1a instead)

Number of:

Intervention sample size

Comparison sample size

Total
sample size

Total response rate

Intervention response rate

Comparison response rate

Clusters







Clusters: At beginning of study

1a

1b

1c (=1a + 1b)

N/A

NA

N/A

Clusters: Contributed at least one individual at baseline

2a

2b

2c (=2a + 2b)

=2c/1c

=2a/1a

=2b/1b

Clusters: Contributed at least one individual at first follow-up (timing)

3a

3b

3c (=3a + 3b)

=3c/1c

=3a/1a

=3b/1b

Clusters: Contributed at least one individual at second follow-up (timing)

4a

4b

4c (=4a + 4b)

=4c/1c

=4a/1a

=4b/1b

Individuals in non-attriting clustersa







Individual: At time that clusters were assigned to condition

5a

5b

5c (=5a + 5b)

N/A

NA

N/A

Individual: Who consented

6a

6b

6c (=6a + 6b)

=6c/5c

=6a/5a

=6b/5b

Individual: Contributed a baseline survey

7a

7b

7c (=7a + 7b)

=7c/5c

=8a/5a

=8b/5b

Individual: Contributed to first follow-up survey (timing)

8a

8b

8c (=8a + 8b)

=8c/5c

=9a/5a

=9b/5b

Individual: Contributed to the impact analysis at first follow-up (timing)

(accounts for item non-response and any other analysis restrictions)b

9a

9b

9c (=9a + 9b)

=9c/5c

=9a/5a

=9b/5b

Individual: Contributed to second follow-up survey (timing)

10a

10b

10c (=10a + 10b)

=10c/5c

=10a/5a

=10b/5b

Individual: Contributed to the impact analysis at second follow-up (timing)

(accounts for item non-response and any other analysis restrictions)b

11a

11b

11c (=11a + 11b)

=11/5c

=11a/5a

=11b/5b

a For all rows in this section, do not include individuals from clusters that dropped (attrited) over the course of the study. For example, if you randomly assigned 10 clusters (5 to each condition), and one intervention group cluster (e.g. school) dropped from the study, you would only include individuals in this section from the 9 clusters that did not drop from the study. Because the cluster-level response rate in the above rows already captures that dropped cluster, you do not need to count individuals from the lost clusters in your individual-level response rates.

b See guidance in section IV.A for defining your analytic sample(s).



Reminder from instructions: For rows that account for item non-response and other analysis restriction, note that you may have very different sample sizes for two outcomes of interest because of very different rates of missing data for the outcomes. If this is the case, please add a row for each outcome in each time period, as needed. Indicate in a table note to which outcome the sample sizes apply. For example, if you have two primary outcomes and there was very different response rates on the items needed to construct these outcomes, you should include two rows for “Contributed to first follow-up (accounts for item non-response and other analysis restrictions)”: one for the analysis sample for outcome one and one for the analysis sample for outcome two.



Table IV.2. Outcome measures used for primary impact analyses research questions (this template includes example data in italics, as an example for you to consider for your own report)

Outcome measure

Description of the outcome measure

Source

Timing of measure

Marriage status

The outcome measure is a yes/no response taken directly from the question in the survey, “Are you currently married?”

Local follow-up survey

6 months after intervention ends

Level of affection

The outcome measure is a scale (value range 1 to 5) calculated from both partners’ responses as the average of five survey items measuring support, intimacy, commitment, trust, and friendship.

Cronbach’s alpha (if applicable): [enter number]

Local follow-up survey

6 months after intervention ends

Notes: [Anything to note about the information above]



Table IV.3. Outcome measures used for secondary impact analyses research questions (this template includes example data italics, as a sample for you to consider for your own report)

Outcome measure

Description of outcome measure

Source

Timing of measure

Level of affection

The outcome measure is a scale (value range 1 to 5) calculated from both partners’ responses as the average of five survey items measuring support, intimacy, commitment, trust, and friendship.

Cronbach’s alpha (if applicable): [enter number]

nFORM exit survey

At post-test (immediately after intervention ends)

Notes: [Anything important to note about the information above]



Table IV.4. Summary statistics of key baseline measures and baseline equivalence across study groups, for individuals/couples completing [Outcome measure at survey follow-up period]

Baseline measure

Intervention mean (standard deviation)

Comparison mean (standard deviation)

Intervention versus comparison mean difference

(p-value of difference)

Effect size

(optional)

Female (%)





Race/ethnicity (%)





Hispanic





Non-Hispanic White





Non-Hispanic Black





Non-Hispanic Asian





Socioeconomic status





Outcome measure 1

(range: 1 to 5)





Outcome measure 2





Sample size





Notes: p-values are include in parentheses. Effect sizes are calculated by dividing the differences in means by the standard deviation of the comparison group. [Anything else important to note about the information above]



Reminder from instructions: Please present an equivalence table for each analytic sample (sample on which impacts are estimated) being used to answer the primary research questions.

Table V.1. Covariates included in impact analyses

Covariate

Description of the covariate

Age

Age (in years) as of the baseline data collection

Baseline marital status

Marital status (1 = married; 0 = not married) as of the baseline data collection


Notes: [Anything to note about the analysis.]





















































Table V.2. Post-intervention estimated effects using data from [Survey follow-up time period] to address the primary research questions

Outcome measure

Intervention mean or % (standard deviation)

Comparison mean or % (standard deviation)

Intervention compared to comparison mean difference

(p-value of difference)

Effect size

(optional)

Outcome 1





Outcome 2















Sample Size





Source: [Name for the Data Collection, Date. For instance, first follow-up surveys administered 12 months after the program.]

Notes: p-values are included in parentheses. Effect sizes are calculated by dividing the differences in means by the standard deviation of the comparison group. [Anything to note about the analysis. See Table IV.2 for a more detailed description of each measure and Chapters IV and V.B for a description of the impact estimation methods]



Table V.3. Differences in means between intervention and comparison groups estimated using alternative methods

Outcome

Benchmark approach

No covariate adjustment

Name of sensitivity approach 2

Name of sensitivity approach 3

Primary Research Questions

Outcome 1





Outcome 2





Outcome 3










Source: [Name for the Data Collection, Date. For instance, Follow-up surveys administered six to eight months after the program.]

**/*/+ Differences are statistically significant at the .01/.05/.10 levels, respectively.

Notes: [Anything to note about the analysis]





Table V.4. Post-intervention estimated effects using data from [Survey follow-up time period] to address the secondary research questions

Outcome measure

Intervention mean or % (standard deviation)

Comparison mean or % (standard deviation)

Intervention compared with comparison mean difference

(p-value of difference)

Effect size

(optional)

Outcome 1





Outcome 2















Sample Size





Source: [Name for the Data Collection, Date. For instance, first follow-up surveys administered 12 months after the program.]

Notes: p-values are included in parentheses. Effect sizes are calculated by dividing the differences in means by the standard deviation of the comparison group. [Anything else to note about the analysis. See Table IV.3 for a more detailed description of each measure and Chapters IV and IV.B for a description of the impact estimation methods.]
























VIII. APPENDIX TABLES AND FIGURES TO SUPPLEMENT FINAL IMPACT REPORT


Appendix A: Logic Model (if applicable)

[Paste logic model here]

































Appendix B: Data, Sample, and Measures (if applicable)

Table B.1. Data used to address implementation research questions

Implementation element

Research question

Data source

Timing/frequency of data collection

Party responsible for data collection

Fidelity

Were all intended intervention components offered and for the expected duration?

Workshop sessions in nFORM

All sessions delivered

Intervention staff

Fidelity

What content did the clients receive?

Fidelity tracking log or protocol; attendance logs; session observations

Every session for fidelity tracking and attendance logs; two times a year for session observations

Intervention staff for fidelity tracking and attendance logs; study staff for session observations

Fidelity

Who delivered services to clients?

Staff applications; hiring records; training logs

One time X months after start of implementation; annually

Intervention staff

Fidelity

What were the unplanned adaptations to key intervention components?

Adaptation request; work plan; 6-month progress report; annual progress report

Annually; ad hoc

Intervention staff; study staff

Dosage

How often did clients participate in the intervention on average?

Workshop sessions and individual service contacts in nFORM; attendance logs

All sessions delivered

Intervention staff

Quality

What was the quality of staffparticipant interactions?

Observations of interaction quality, using protocol developed by study staff

X percentage of sessions selected at random for observation

Study staff

Engagement

How engaged were clients in the intervention?

Observations of engagement, possibly using an engagement assessment tool; ratings from facilitator fidelity logs; engagement ratings from participant satisfaction surveys

Y percentage of sessions selected at random for observation

Study staff

Context

What other HM/RF programming was available to study participants?

Interviews with staff from partnering agencies in the community; survey items on baseline and follow-up assessments; websites of other agencies in the community providing HM/RF programming

Once a year; ad hoc

Study staff

Context

What external events affected implementation?

Interviews with community/county representatives; list of site/school closures

Once a year; ad hoc

Study staff

Note: We use “clients” in the questions above to avoid redundancy.

Table B.2. Key features of the impact analysis data collection


Data source

Timing of data collection

Mode of data collection

Party responsible for data collection

Start and end date of data collection

Intervention

Intervention group study participants

Enrollment (baseline)

End of intervention (8 months after enrollment)


In-person online survey (nFORM)


Program staff

September 2016 through January 2020



3 months after the end of the intervention (11 months after enrollment)

6 months after the end of the intervention (14 months after enrollment)

Telephone survey

Evaluation staff

August 2017 through March 2021

Counterfactual

Comparison group study participants

Enrollment (baseline)

8-month follow-up

In-person online survey (nFORM)

Program staff

September 2016 through January 2020



11-month follow-up

14-month follow-up

Telephone survey

Evaluation staff

August 2017 through March 2021

























[Paste updated CONSORT diagram from you Analysis Plan here.]



































Appendix C: Compromised RCT, Attrition, and/or Baseline Equivalence of original RCT sample (if applicable)



C.1 If your study was originally an RCT but had to construct equivalent groups using a QED approach because of high attrition, lack of baseline equivalence, and/or another issue that compromised the random assignment, please describe the details in this appendix. Table C.1 can be used to summarize baseline equivalence of the RCT sample.



Table C.1. Baseline equivalence of the original RCT intervention and control groups

Baseline measure

Intervention mean (standard deviation)

Comparison mean (standard deviation)

Intervention versus comparison mean difference

(p-value of difference)

Effect size

Female (%)





Race/ethnicity (%)





Hispanic





Non-Hispanic White





Non-Hispanic Black





Non-Hispanic Asian





Outcome measure 1

(range: 1 to 5)





Outcome measure 2





Sample size





Notes: p-values are include in parentheses. Effect sizes are calculated by dividing the differences in means by the standard deviation of the comparison group. [Anything else important to note about the information above]



C.2. For both QEDs and RCTs that had to construct equivalent groups, please describe the approach to constructing equivalent groups in this appendix.

7

File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
AuthorWilliam Garrett
File Modified0000-00-00
File Created2023-10-29

© 2024 OMB.report | Privacy Policy