Instrument 9: Report template for descriptive evaluations

THE PAPERWORK REDUCTION ACT OF 1995

This collection of information is voluntary and will be used to provide the Administration for Children and Families with information to help refine and guide SRAE program development. Public reporting burden for the collection of information is estimated to average 32 hours per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #: XXXX-XXXX, Exp: XX/XX/20XX. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, Susan Zief [email protected]

INSTRUMENT 9: SRAE Descriptive Report Template

[Feel free to add graphic or logo]

Evaluation of
[Program Name(s)] in
[Geographic Area]

Final Implementation and Outcomes Report for

[Grantee Organization]

[Date]

Prepared by

[Evaluator Organization Authors]

Recommended Citation: [Authors]. (Year). Evaluation of [Program Name(s)] in [Geographic Area]. Evaluator location: evaluator organization.

Acknowledgments:

[In this space, please list contributors to this evaluation report, including reviewers and editors that you would like to acknowledge; feel free to acknowledge any persons critical in making the evaluation possible, the program implementation possible, and so on.]

Disclosure:

[In this space, please disclose any conflict of interest, financial or otherwise.]

This publication was prepared under Grant Number [Insert Grant Number] from the Family and Youth Services Bureau within the Administration for Children and Families (ACF), U.S. Department of Health & Human Services (HHS). The views expressed in this report are those of the authors and do not necessarily represent the policies of HHS, ACF, or the Family and Youth Services Bureau.

[Copy from your preexisting abstract, as available. Note: The methods, implementation findings, and outcomes findings sections are new and should be completed as part of this report.]

Evaluation abstract: “The Evaluation of [Intervention Name] ] in [Geographic Area]”

Grantee

Grantee name:

Project lead:

Email address:

Evaluator

Evaluator’s organization:

Evaluator lead:

Email address:

Intervention name

[Start writing here.]

Intervention description

[Start writing here.]

Implementation evaluation

[Start writing here.]

Outcomes evaluation

[Start writing here.]

Sample and setting

[Start writing here.]

Research design

[Start writing here.]

Data collection

[Start writing here.]

Methods

[Start writing here.]

Implementation findings

[Start writing here.]

Outcomes findings

[Start writing here.]

Conclusion

[Start writing here.]

THE Evaluation of [Program name(s)] in [geographic area]

I. Introduction

A. Introduction and study overview

[Copy and paste text/Start writing here.]

B. Study objectives

[Copy and paste text/Start writing here.]

II. Description of Youth served

[Provide an overview of the section here.]

A. Overview of youth participating in evaluation

[Copy and paste text/Start writing here.]

B. Needs assessment activities

[Copy and paste text/Start writing here.]

C. Youth Recruitment Strategy

[Copy and paste text/Start writing here.]

III. Programming

[Provide an overview of the section here.]

A. Description of program

[Copy and paste text/Start writing here.]

B. Description of planned adaptations

[Copy and paste text/Start writing here.]

IV. Study sample formation

[Copy and paste text/Start writing here.]

V. Implementation evaluation design

[Provide an overview of the section here.]

A. Research questions

[Copy and paste text/Start writing here.]

B. Study sample and data collection

[Copy and paste text/Start writing here.]

C. Measures and methods

[Copy and paste text/Start writing here.]

VI. Outcomes evaluation design

[Provide an overview of the section here.]

A. Research questions

[Copy and paste text/Start writing here.]

B. Research design

[Copy and paste text/Start writing here.]

C. Data collection

[Copy and paste text/Start writing here.]

D. Measures

[Copy and paste text/Start writing here.]

[Copy and paste Table VI.1 here.]

E. Analytic sample

[Copy and paste text/Start writing here.]

[Copy and paste Table VI.2 here.]

F. Participant characteristics

[Copy and paste text/Start writing here.]

[Copy and paste Table VI.3 here.]

G. Methods

[Copy and paste text/Start writing here.]

VII. Evaluation findings

A. Implementation evaluation: findings

[Copy and paste text/Start writing here.]

[Copy and paste Table VII.1 here.]

B. Outcomes evaluation: findings

[Copy and paste text/Start writing here.]

[Copy and paste Table VII.2 here.]

VIII. Conclusion

A. Summary

[Copy and paste text/Start writing here.]

B. Limitations

[Copy and paste text/Start writing here.]

C. Discussion

[Copy and paste text/Start writing here.]

IX. References

[Copy and paste text/Start writing here.]

Appendix A: Logic Model

[Paste logic model here]

Appendix B: Implementation Evaluation

[Copy and paste text/Start writing here.]

[Copy and paste Table B.1 here.]

Appendix C: Data Cleaning

[Copy and paste text/Start writing here.]

[Copy and paste Table C.1 here.]

Appendix D: Implementation Evaluation Methods

[Copy and paste text/Start writing here.]

Appendix E: Map of Service Region

[Copy and paste text/Start writing here.]

Appendix S: Sensitivity Analyses

[Copy and paste text/Start writing here.]

[Copy and paste Table S.1 here.]

Table VI.1. Outcome measures used for the research questions in the outcomes analyses (this template includes example data in italics, as a sample for you to consider for your own report)

Name of outcome measure	Source item(s)	Constructed measure	Timing of measure relative to program
Recent sexual intercourse	Have you had recent sexual intercourse?	Dichotomous variable coded as 1 if answered yes; zero if no OR ever had sex = no; and missing otherwise.	6 months after program ends

Table VI.2a. Sample sizes and response rates for youth

Number of youth	Time period	Sample size	Response rate
Consented to participate		1c	n.a.
Completed a baseline survey		2c	= 2c/1c
Completed a follow-up survey	Immediately post-programming	3c	= 3c/1c
Contributed to the outcomes analysis at follow-up (accounts for item nonresponse)a	Immediately post-programming	4c	= 4c/1c
Completed a follow-up survey	6 months post-programming	5c	= 5c/1c
Included in the outcomes analysis at follow-up (accounts for item nonresponse)a	6 months post-programming	6c	= 6c/1c

^a See guidance in Section VI.E for defining your analytic sample(s).

n.a. = not applicable.

Table VI.2b. Youth and [cluster] sample sizes

Number of:	Baseline	[Survey time period]
Youth
Enrolled at time of survey
Completed survey
Completed both surveys OR enrolled at both time points	n.a.
[Cluster]
Number of clusters
Average number of youth per cluster
Range in number of youth per cluster

n.a. = not applicable.

Table VI.3a. Characteristics of the youth completing [Survey follow-up period], compared with youth who did not respond (Only use for studies that can link individuals’ pre-test and post-test data)

Baseline measure	Youth who responded to follow-up survey	Youth who did not respond to follow-up survey	Difference between responders and nonresponders
	Percentage or mean (standard deviation)	Percentage or mean (standard deviation)	p-value
Age or grade level
Gender (female)
Race/ethnicity
Category 1
Category 2
Category 3
Category 4
Outcome Measure 1
Outcome Measure 2
Outcome Measure 3
Outcome Measure 4
Sample size			n.a.

n.a. = not applicable

Table VI.3b. Characteristics of the youth completing baseline survey (Only use for studies that cannot link individuals’ pre-test and post-test data)

Baseline measure	Percentage or mean (standard deviation)
Age or grade level
Gender (female)
Race/ethnicity
Category 1
Category 2
Category 3
Category 4
Sample size

Notes: See Table VII.2 for a presentation of baseline means of the outcome measures.

Table VII.1. Targets and findings for each measure used to answer implementation evaluation research questions (Note: Example data included in italics. Please remove before completing the table)

Implementation element	Research question	Measure	Target	Results
Fidelity	Were all intended program components offered, and for the expected duration?	Total number of sessions delivered Average session duration, calculated as the average of the recorded session lengths (in minutes)	95 percent of groups to receive all 12 sessions Average session duration will be at least 40 minutes	75 percent of groups received all 12 sessions Average duration of session was 35 minutes
Fidelity	What content did the youth receive?	Total number of topics covered, calculated as the average of the total number of topics checked by each program facilitator in the daily fidelity tracking log or protocol	95 percent of groups to receive instruction on 90 percent of the topics	65 percent of groups were taught 90 percent of the topics; 45 percent of groups were taught 100 percent of the topics
Fidelity	Who delivered services to youth?	Number and type of staff delivering services to study participants, such as the number of session facilitators Percentage of staff who received minimum training, calculated as the number of staff who received at least 20 hours of training divided by the total number of staff who delivered the program	Three full-time health educators will deliver programming All health educators to receive at least 20 hours of training each year	A total of five staff were employed during evaluation to fill three full-time health educator positions 4 of 5 educators received at least 20 hours of training each year (average = 24.5 hours)
Fidelity	What were the unplanned adaptations to key program components?	List of unplanned adaptations, such as a change in setting, sessions added or deleted, and components cut	n.a.	45 percent of educators skipped at least one component in Lessons 3 and 5
Dosage	How often did youth participate in the program on average?	Average number (or percentage) of sessions youth attended Percentage of the sample attending the required or recommended proportion of sessions Percentage of the sample that did not attend sessions at all	n.a. 75 percent of youth to attend 75 percent of the program sessions Less than 5 percent of the sample gets none of the program	Youth attended 8 sessions on average 60 percent of youth attended 75 percent of the program sessions 10 percent of the sample received none of the program
Quality	What was the quality of staff–participant interactions?	Percentage of observed sessions with high quality interactions, calculated as the percentage of observed interactions that study staff scored as “high quality”	90 percent of observed sessions to be implemented with high quality (rated as a 3.5 out of 4 on the quality scale)	87 percent of observed sessions implemented with high quality (rated as a 3.5 out of 4 on the quality scale)
Engagement	How engaged were youth in the program?	Percentage of observed sessions with moderate participant engagement, calculated as the percentage of sessions in which study staff scored participants’ engagement as “moderately engaged” or higher	90 percent of observed sessions to be implemented with moderate to high engagement	85 percent of observed sessions implemented with moderate to high engagement
Context	What other pregnancy prevention programming was available to study participants?	Percentage of the sample receiving pregnancy prevention programming from other providers, constructed from immediate post-survey data on experiences outside of the current program	Less than 20 percent of youth to receive formal content outside of the program	35 percent of youth (50 percent in control group and 15 percent in treatment group) received other pregnancy prevention programming
Context	What external events affected implementation?	Percentage and total number of sessions not delivered due to event in the community, if any	n.a.	Hurricane in community closed some programming sites for a week. Sessions were made up for 60 percent of youth in those sites.

n.a. = not applicable

Table VII.2. Changes in outcomes using data from pre-test and [Survey follow-up time period]

Outcome measure	Pre-test percentage or mean (standard deviation)	[Survey time period] percentage or mean (standard deviation)	Difference between means (p-value)
Outcome 1
Outcome 2
Outcome 3
Outcome 4
Sample size

Source: [Name for the data collection, date. For instance, Follow-up surveys administered 6 to 8 months after the end of the program.]

Notes: [Anything to note about the analysis. See Table VI.1 for a more detailed description of each measure, and Chapter VI for a description of the methods used to assess program outcomes.]

Table B.1. Data used to address implementation research questions (Note: Example data included in italics. Please remove before completing the table)

Implementation element	Research question	Measure	Data collection frequency/sampling	Sample
Fidelity	Were all intended program components offered for the expected duration?	Total number of sessions delivered Average session duration, calculated as the average of the recorded session lengths (in minutes)	All sessions delivered are captured in MIS Session length sampled weekly	All program sessions All program sessions
Fidelity	What content did the youth receive?	Total number of topics covered, calculated as the average of the total number of topics checked by each program facilitator in the daily fidelity tracking log or protocol	Content from all sessions is captured in MIS	All program sessions
Fidelity	Who delivered services to youth?	Number and type of staff delivering services to study participants, such as the number of session facilitators Percentage of staff who receive minimum training, calculated as the number of staff who received at least 20 hours of training divided by the total number of staff who delivered the program	Staff records Training attendance records from all training activities are captured in MIS	All program staff All program staff
Fidelity	What were the unplanned adaptations to key program components?	List of unplanned adaptations, such as a change in setting, sessions added or deleted, and components cut	As needed	All program sessions
Dosage	How often did youth participate in the program on average?	Average number (or percentage) of sessions youth attended Percentage of the sample attending the required or recommended proportion of sessions Percentage of the sample that did not attend sessions at all	Student attendance at all sessions is captured in MIS Student attendance at all sessions is captured in MIS Student attendance at all sessions is captured in MIS	All program sessions All program sessions All program sessions
Quality	What was the quality of staff–participant interactions?	Percentage of observed sessions with high quality interactions, calculated as the percentage of observed interactions that study staff scored as “high quality”	Convenience sample of 10 percent of classroom sessions were selected for observation	10 percent of sessions
Engagement	How engaged were youth in the program?	Percentage of observed sessions with moderate participant engagement, calculated as the percentage of sessions in which study staff scored participants’ engagement as “moderately engaged” or higher	Random sample of 5 percent of classroom sessions were selected for observation	All program sessions
Context	What other pregnancy prevention programming was available to study participants?	Percentage of the sample receiving pregnancy prevention programming from other providers, constructed from immediate post-survey data on experiences outside of the current program	Post-program	All youth in the program
Context	What external events affected implementation?	Percentage and total number of sessions not delivered because of some other event in the community, if any	As needed	All program sessions

Table C.1. Participant characteristics with and without nonresponse weights, by survey time period

Measure	Pre-test means			[Survey time period] means
	Full sample	Weighted complete-case sample	Complete-case sample	Full sample	Weighted complete-case sample	Complete-case sample
Age or grade level
Gender (female)
Race/ethnicity
Category 1
Category 2
Category 3
Category 4
Measure 1
Measure 2
Measure 3
Measure 4
Sample Size

Source: [Name for the data collection, date. For instance, Follow-up surveys administered 6 to 8 months after the end of the program.]

Notes: [Anything to note about the analysis. See Table VI.1 for a more detailed description of each measure, and Chapter VI for a description of the methods used to assess program outcomes.]

Table S.1. Sensitivity of analyses using data from [Survey follow-up period] to address the outcomes study research questions

Intervention compared with comparison	Benchmark approach: mean difference	Benchmark approach: p-value	Name of Sensitivity Approach 1: mean difference	Name of Sensitivity Approach 1: p-value
Outcome 1
Outcome 2
Outcome 3
Outcome 4

Source: [Name for the data collection, date. For instance, Follow-up surveys administered 6 to 8 months after the end of the program.]

Notes: [Anything to note about the analysis. See Table VI.1 for a more detailed description of each measure, and Chapter VI for a description of the methods used to assess program outcomes.]