PREIS impact Report Guidance
A key goal of the Family and Youth Services Bureau (FYSB) Personal Responsibility Education Innovative Strategies (PREIS) Program is to ensure that rigorous evidence on program effectiveness contributes to the knowledge base on adolescent pregnancy prevention.
This document guides you on structuring a final report to be comprehensive and accessible. The goal of the final report is to document your evaluation and share the findings to a public audience. In hopes of saving time when completing this template, many of the report sections draw directly from the material prepared for the evaluation abstract, and Impact and Program Implementation Evaluation Analysis Plans. You can use text from those sources to simplify report writing. Here, we provide an annotated outline that contains the following guidance for each section: (1) “Purpose” describes the purpose of the section and what you should discuss, (2) “Instructions and reminders” discusses things for you to keep in mind when writing the section, (3) “Potential resources” lists existing documents that might be potential sources for you to write the section, and (4) “Non-text elements” guides whether non-text elements (tables or figures) could be useful.
Write your report using the accompanying template, which we set up to make the production and 508-compliance process easier and facilitate review by FYSB and the RETA team.1 An attached Word file (PREIS_Impact Report Template.docx) provides an outline of the report with places for you to fill in each section. When filling out the template, please use complete sentences. A separate attached table shell file (PREIS_Impact Report Tables.docx) provides some required and optional table shells for you to use and paste into the outline file as you write the report. Using these shells will enable FYSB to more quickly make the reports 508 compliant. Both files are also saved on SharePoint in the Templates folder.
Here are some additional suggestions for your impact report:
Organize the final report so that it is about 30 to 40 double-spaced pages (15 to 20 single-spaced pages), excluding tables, figures, references, and appendices.
Write the report for all audiences, not just other researchers. Write as if the audience has not been involved in the grant and knows nothing about the program or the evaluation. The report should provide enough detail for readers to understand the program and its evaluation, and it should be free of project- or program-specific jargon and abbreviations.
Reach out to your RETA with questions about the report guidance or your approach as you begin to work on your report. Getting questions resolved early in the process will simplify the review process at the end of the grant period.
Please submit a report that you believe is ready for external review, not one to be read as a draft. It will have ideally been edited and read by multiple people before you submit it to minimize the amount of editorial comments your Federal Project Officer (FPO) and RETA will have to provide. Their goal is to focus on content and technical details rather than formatting and editorial changes.
Please email your final report as Word files (not PDF) to your FPO and copy your RETA liaison by [due date]. For consistency, please use this common naming convention when submitting your report: [Grantee Name] Impact Report_ [report draft date]. Your FPO and RETA liaison will review the final report, provide comments and suggested edits, and return it to you for revisions. Your FPO must approve your final report by the end of your grant period.
Cover Page
The cover page for your report should include the title of the report and all authors.
Disclose any conflicts of interest—financial or otherwise—on the cover page. For an example of how to identify a conflict of interest, please see the website of the International Committee of Medical Journal Editors. For example, if someone on the evaluation team received a grant from an organization included in the evaluation, you could include language such as “[insert name] reports receipt of a grant from [organization name] during the evaluation period.” Please note that an evaluation team that is not completely independent from the program team (that is, if they are not different organizations with completely separate leadership and oversight) is a conflict of interest you must document.
Finally, the cover page should include the attribution to FYSB:
This publication was prepared under Grant Number [Insert Grant Number] from the Family and Youth Services Bureau within the Administration for Children and Families (ACF), U.S. Department of Health and Human Services (HHS). The views expressed in this report are those of the authors and do not necessarily represent the policies of HHS, ACF, or the Family and Youth Services Bureau.
Evaluation abstract
Purpose |
Provide a one- to two-page executive summary of the final report. |
Instructions and reminders |
Please complete the abstract in the outline template. You can copy and paste information for most of the fields from your most recently approved abstract, although you should remember to update the abstract if anything has changed since its approval. There are some additional fields to include, specifically the Methods and Findings sections. You can find your most recently approved evaluation abstract on SharePoint in your Abstract folder. |
Potential sources |
|
Non-text elements |
None |
I. Introduction
A. Introduction and study overview
Purpose |
Orient the reader to the study. |
Instructions and reminders |
In this section, explain (1) the need for teen pregnancy prevention for the particular population (defined by locality, age, gender, race/ethnicity, etc.) studied; (2) the rationale for selecting your program and how the evaluation fits within the FYSB PREIS grant program; (3) previous research describing the effects of the program, including, if applicable, how prior findings were assessed by the Evidence Review; and (4) that this report describes the implementation and impact of the PREIS-funded program. The reader should understand why the program was targeted to certain youth and the motivation for selecting the chosen program. |
Potential sources |
|
Non-text elements |
None |
B. Primary research questions
Purpose |
Articulate the key research questions about the impacts of the program on sexual behavior outcomes of youth. |
Instructions and reminders |
This section should present the primary research questions. Remember that the primary research questions should focus on the impact of the intervention (that is, the specific aspects of your program that you test in the evaluation) on at least one behavioral outcome measure. All primary research questions should focus on impacts at a specific time point. The behavioral outcomes and time points should clearly connect to the program’s theory of change. Your research questions should be drawn from previously approved documents (including your evaluation design plan or impact and program implementation evaluation analysis plan), although you may need to make some changes to those if anything has changed. |
Potential sources |
|
Non-text elements |
None |
C. Secondary research questions
Purpose |
Outline additional, important research questions about other potential outcomes of your program. |
Instructions and reminders |
This section should present the secondary research questions. Secondary research questions would include explorations of other outcomes that the intervention might influence or other justifiable explorations of program effectiveness (for example, whether the program works better for certain populations). These research questions might focus on one of several possible types of outcomes:
|
Instructions and reminders (Continued) |
As with the primary research questions, all the secondary research questions should focus on outcomes at a specific time point and should align with the program’s theory of change. For instance, if your theory of change posits that knowledge should change immediately following the end of the program, you should examine the impact on knowledge outcomes at your immediate post-program survey. You may expect to see impacts on some outcomes (like substance use or impulsive behavior) over a longer period of time, so you would want to examine those outcomes at the time of the short or long-term follow-ups. NOTE: If your analysis plan included additional analyses beyond impact analyses, you can include them in a section I.D.: Additional analyses. Grantees might also have research questions for the implementation study (for example, questions about adherence to the program model or youth attendance). These research questions should go in the implementation evaluation section (section IV below) and not this section. You can add a general statement about your implementation questions (for example, that you also assessed program dosage and quality to understand program impacts) and then refer readers to section III.C.1 for details. |
Potential sources |
|
Non-text elements |
None |
II. Programming for treatment and comparison groups
Provide a one- or two-paragraph overview to this section highlighting that it will summarize the intended program and comparison conditions and that section V.A will describe what each group actually received.
A. Description of program as intended
Purpose |
Summarize the program being tested. |
Instructions and reminders |
This section should describe the intervention condition as intended (or what the intervention group was supposed to receive). The implementation results section (Section V.A) should discuss what the intervention group actually received in this study. Discuss (1) program activities; (2) program content; (3) expectations for implementation, including the implementation location or setting (for example, schools or clinics), duration and dosage, and staffing; and (4) theory of change or logic model for the program, including the expected outcomes. Include a discussion of the APS your program covered, how you intended to covered the, and how you selected them to be a focus of your program. A graphical representation of your logic model or theory of change is required as Appendix A in the report. |
Potential sources |
|
Non-text elements |
Readers often find it useful when you present information about the program description in a table (for example, with rows for target population, program activities, program content, planned mode of delivery, and so on). Your logic model must be included in Appendix A. Be sure the logic model includes the APS content and outcomes. |
B. Description of comparison condition as intended
Purpose |
Summarize the counterfactual being examined. |
Instructions and reminders |
This section should describe the comparison condition as intended (or what the comparison group was supposed to receive). The implementation results section (Section V.A) should discuss the actual comparison group experience for the study. If the comparison group was intended to receive services as usual, briefly describe what services or programs were available to youth in your service area that you expected youth in the comparison group might access. If the comparison group received a defined program, describe what the program offered and how it was intended to be implemented. The description of the condition should include the elements listed previously for the program description but does not need to discuss theory of change. If the intervention and comparison conditions share common elements, please highlight the similarities and differences in services received in the text, and consider adding a table to help the reader compare the content. |
Potential sources |
|
Non-text elements |
Table differentiating the intervention and comparison conditions (if applicable) |
III. Impact evaluation design
Provide a one- or two-paragraph overview to this section highlighting that it will summarize the evaluation design, sample, data collection, and analytic methods for the impact evaluation.
A. Identification and recruitment of the study participants
Purpose |
Describe the study setting, context, and eligibility criteria. |
Instructions and reminders |
This section should describe the study’s setting and context, the study’s target population, and how you identified and recruited the sample for the study (both youth and clusters, as appropriate), including any eligibility criteria. Include information on the success of the recruitment process—that is, the number of primary units (for example, schools) contacted and number recruited. This discussion should focus on the eligible and enrolled sample (for example, the 3,000 youth enrolled in 9th grade health classes in two school districts). Section III.E will focus on the final analytic samples. |
Potential sources |
|
Non-text elements |
None |
B. Research design
Purpose |
Summarize the research design used to assess the effectiveness of the intervention. |
Instructions and reminders |
This section should clearly identify the design you used to test program effectiveness. Is it a randomized controlled trial (RCT)? If so, did it involve individual or cluster random assignment? Is it a quasi-experimental design (QED)? How did you form the study groups? Please indicate any limitations of the design or how it was implemented. For example, were there concerns about knowledge of study assignment during the consent process (if consent was gathered after random assignment in a clustered RCT) or concerns that different types of schools volunteered for intervention versus comparison in a QED? Consider addressing relevant issues that you have discussed with your FPO and RETA over the course of the study. |
Instructions and reminders (Continued) |
Note on using matching: Following best practice in the field, if your study design is an RCT with high attrition at the unit of assignment, and there is a lack of baseline equivalence in the analytic sample, you should use a matching analysis to create a more rigorous comparison group. In this scenario, you should discuss the fact that the study was originally an RCT, but, because of high attrition and lack of baseline equivalence (which you will document in an appendix), the study had to construct matched comparison groups. If you need assistance with attrition calculations or determining whether you should use matching, please reach out to your RETA. Similarly, if the study is a QED with a lack of baseline equivalence in the analytic sample, the study should use a matching analysis to better account for the baseline differences in the groups and create a more rigorous comparison group. In these scenarios, present the matching analysis as the main or benchmark analysis in the report, with additional specifications as sensitivity analyses. Include any additional details regarding the matching analysis or intent-to-treat specification in an appendix. |
Potential sources |
|
Non-text elements |
None |
C. Data collection
Purpose |
Indicate how data on outcomes of interest as well as key explanatory variables, including baseline assessments and demographics were obtained from sample members |
Instructions and reminders |
Describe the data collections conducted, including timing, mode of administration, and overall process, and include information on incentives. Provide details on differences between intervention and comparison conditions in timing, mode, and processes used for data collection, as appropriate. |
Potential sources |
|
Non-text elements |
It may be useful to present a table to summarize the information. |
D. Measures
Purpose |
Describe how the outcomes of interest in the primary and secondary research questions were operationalized using survey data or other data elements. |
Instructions and reminders |
Define the outcomes that the primary and secondary research questions examine. Briefly explain how you operationalized and constructed each outcome measure. If you constructed a measure from multiple items, please document the source items and explain how you did so to create an outcome for analysis. If a detailed description of measure construction is necessary, please include that information in an appendix. In this section, only present information on the outcomes that are the focus of the primary or secondary research questions. |
Potential sources |
|
Non-text elements |
Please present this information in a table. Please refer to the table shells document, which includes 508-compliant (landscape) table shells for Tables III.1 (primary research questions) and III.2 (secondary research questions). Do not include Table III.2 if there are no secondary research questions (Note: in the template, Table III.1 includes examples for you). Instructions for completing Tables III.1 and III.2
|
E. Study sample
Purpose |
Describe the flow of participants into the sample. |
Instructions and reminders |
Use this section to describe how you created the analytic samples—that is, the flow of sample members from the point of random assignment (or consent for QEDs) through the follow-up assessments used in the primary and secondary research questions, factoring in non-consent, attrition, and item nonresponse. For studies with cluster-level assignment, this section should include (1) the total number of clusters randomized (or consented for QEDs) and the number of youth in those clusters; (2) the number of students for whom parental consent was obtained after random assignment (if applicable); (3) the number of clusters that contributed data at baseline and follow-up (if entire clusters were lost, please document sample loss in this section); (4) the number of youth for whom baseline and follow-up data were obtained for each key measure (follow-up is the time period indicated in the research questions for assessing program effectiveness); and (5) the characteristics of the study sample (the youth and clusters), such as time period, total sample size, sample size by study group, and response rates, for the total sample and by study group. For studies with individual-level assignment, this section should include (1) the total number of youth randomized (or consented for QEDs); (2) the number of students for whom parental consent was obtained after random assignment (if applicable); (3) the number of youth for whom baseline and follow-up data were obtained for each key measure (follow-up is the time period indicated in the research questions for assessing program effectiveness); and (4) the characteristics of the study sample, such as time period, total sample size, sample size by study group, and response rates, for the total sample and by study group. For all studies, this section should indicate sample sizes for your analytic samples (that is, the sample(s) on which you estimate impacts). For example, suppose that the primary research question focuses on a 12-month follow-up, the secondary research question focuses on a 24-month follow-up, and recent sexual activity is the key outcome on which you are assessing program impacts. In this case, you might have two analytic samples: (1) the sample responding to the 12-month follow-up with non-missing data on recent sexual activity (primary analytic sample) and (2) the sample responding to the 24-month follow-up with non-missing data on recent sexual activity (secondary analytic sample). If you are creating an analytic sample for a particular time point when there are multiple behavioral outcomes to be examined (with some item non-response across the outcomes), the RETA team recommends identifying a single, common analytic sample that does not have missing data across all of the outcomes of interest. Using a single, common analytic sample will produce an easy-to-follow presentation of the analyses across multiple outcome measures. If, however, there is substantial item non-response across two or more outcomes, then the RETA team recommends considering each outcome as requiring its own unique analytic sample (baseline equivalence will have to be demonstrated separately for each analytic sample). |
Potential sources |
|
Non-text elements |
As support for the discussion above, include one of the sample flow tables that follow. Use either the cluster-level (Table III.3a) or individual-level (Table III.3b) design table, whichever is appropriate for the study design. The next two pages include more detailed instructions for completing these tables. |
Detailed instructions for TABLEs III.3a and III.3b
Please refer to the table shells document for two versions of 508-compliant table shells for reporting sample flow for either cluster-level (Table III.3a) or individual-level (Table III.3b) assignment designs. Complete only one table for this section. Please use the table appropriate for the study design.
Instructions for completing Table III.3a (For studies with cluster-level assignment)
The purpose of this table is to clearly present the sample sizes and response rates for both clusters and youth for cluster-level assignment studies.
Italicized text highlights how to calculate total sample sizes and response rates given other information in the table.
The table is split into two sections: the top section focuses on cluster sample sizes and response rates and the bottom section focuses on youth sample sizes and response rates.
In the column “Time period,” describe when you administered each survey relative to the end of programming. (Example text is shown in this column in the template. You should include a row for each outcome at each survey time period that you include in your primary or secondary analyses.)
In the columns “Total sample size,” “Intervention sample size,” and “Comparison sample size,” do the following:
In the “clusters” section, enter the number of clusters that were assigned to condition in the “Clusters: at beginning of study” row. In the next four rows, enter the number of clusters in which at least one youth completed the relevant survey.
In the “youth” section, start with the number of youth from non-attriting clusters. For all rows in this section, exclude youth in clusters that dropped (attrited) from the study. For example, if you randomly assigned 10 clusters (5 to each condition) and 1 intervention group cluster dropped from the study, you would only include youth from the 9 clusters that did not drop from the study.
For each row, the value entered in the “Total sample size” column should be the sum of the “Intervention sample size” and “Comparison sample size” (this calculation is shown in italics).
List how many youth were present in the clusters at the time of assignment in the “Youth: At time that clusters were assigned to condition” row.
In the row “Youth: who consented,” enter the number of youth who consented to participate. If consent occurred before assignment, delete this row, and add a note at the bottom of the table indicating that consent occurred before random assignment.
In the rows “Youth: Completed a baseline survey”, and “Youth: completed a follow-up survey”, then list how many youth completed the relevant surveys.
It is likely that missing data on the outcomes of interest or covariates due to item-level non-response will result in slightly different sample sizes than the number of youth who completed the relevant follow-up survey. In the row “Youth: Included in the impact analysis sample at follow-up (accounts for item non-response),” enter the number of youth who are ultimately included in your impact analyses. If your analytic sample sizes vary for different outcomes (because of different rates of missing data), then add a row for each outcome in each time period as needed. Indicate in a table note the outcomes to which the sample sizes apply. For example, if you have two primary outcomes (pregnancy and unsafe sex) and there were different response rates on the items needed to construct these outcomes, you should include two rows for “Youth: Included in o the impact analysis sample at follow-up (accounts for item non-response)”—one for the analysis sample for the pregnancy outcome and one for the analysis sample for the unsafe sex outcome.
In the columns “Total response rate,” “Intervention response rate,” and “Comparison response rate,” please conduct the calculations indicated by the italicized formula.
Note that for the “clusters” section, the denominator for the response rate calculations will be the numbers entered in sample size columns in the “Clusters: At beginning of study” row. For the “youth” section, the denominator for the response rate calculations will be the numbers entered in the sample size columns in the “Youth: At the time that clusters were assigned to condition” row.
To present findings from more than two follow-up periods, please add rows as needed.
If the study includes more than two conditions, please add columns as needed.
Instructions for completing Table III.3b (for studies with individual-level assignment)
The purpose of this table is to clearly present the sample sizes and response rates for youth in individual-level assignment studies
Italicized text highlights how to calculate total sample sizes and response rates given other information in the table.
In the column “Time period,” describe when you administered the survey relative to the end of programming (example text is shown in this column in the template).
In the columns “Total sample size,” “Intervention sample size,” and “Comparison sample size,” enter the number of youth who were assigned to condition in the “Assigned to condition” row. In the “Completed a baseline survey” and “Completed a follow-up survey” rows that follow, enter the number of youth who completed the relevant survey.
For each row, the value entered in the “Total sample size” column should be the sum of the “Intervention sample size” and “Comparison sample size” (this calculation is shown in italics).
In the columns “Total response rate,” “Intervention response rate,” and “Comparison response rate,” please conduct the calculations indicated by the italicized formula.
Note: the denominator for the response rate calculations will be the numbers entered in sample size columns in the “Assigned to condition” row.
It is likely that missing data on the outcomes of interest or covariates due to item-level non-response will result in slightly different sample sizes than the number of youth who completed the relevant follow-up survey. In the row “Youth: Included in the impact analysis sample at follow-up (accounts for item non-response),” enter the number of youth who are ultimately included in your impact analyses. If your analytic sample sizes vary for different outcomes (because of different rates of missing data), then add a row for each outcome in each time period as needed. Indicate in a table note the outcomes to which the sample sizes apply. For example, if you have two primary outcomes (pregnancy and unsafe sex) and there were different response rates on the items needed to construct these outcomes, you should include two rows for “Youth: Included in o the impact analysis sample at follow-up (accounts for item non-response)”—one for the analysis sample for the pregnancy outcome and one for the analysis sample for the unsafe sex outcome. To present findings from more than two follow-up periods, please add rows as needed.
If the study includes more than two conditions, please add columns as needed.
F. Baseline equivalence and sample characteristics
Purpose |
Provide information on how you assessed baseline equivalence for the analytic samples and present the results of the assessment. Describe the sample characteristics for the reader. |
Instructions and reminders |
Briefly describe the methods you used to assess the equivalence of the analytic samples that served to answer the primary and secondary research questions. The analytic method used to show baseline equivalence should account for the study design (for example, clustering or stratification). Include equations for estimating equivalence of analytic samples in the appendix if necessary. Present an equivalence table for each analytic sample (that is, the sample on which effects are estimated) used to answer the primary and secondary research questions. For example, suppose that the primary research question focuses on a 12-month follow-up assessment, the secondary research question focuses on a 24-month follow-up assessment, and recent sexual activity is the key outcome on which you are assessing program impacts. In this case, provide tables for (1) the sample responding to the 12-month follow-up with non-missing data on recent sexual activity (primary analytic sample) and (2) the sample responding to the 24-month follow-up with non-missing data on recent sexual activity (secondary analytic sample). See the note in section III.E on establishing analytic samples when item nonresponse varies substantially across outcomes of interest. The baseline equivalence tables must include demographic characteristics (age or grade, gender, and race and ethnicity) and a measure of the outcome of interest assessed at the baseline. For each group, the table should document (1) sample sizes for each characteristic reported; (2) mean and standard deviation for continuous variables or the proportion, as a decimal, for categorical variables; (3) the difference in means (or proportions) between the two groups; and (4) the p-value of the difference between the two groups. This presentation is similar to the Excel baseline equivalence table you have completed in the past. |
Instructions and reminders (Continued) |
Include a narrative description of the sample characteristics for the reader (for example, the age and gender of the sample). |
Potential sources |
|
Non-text elements |
Please refer to the table shells document, which includes a 508-compliant (landscape) table shell for Table III.4 to be used to demonstrate baseline equivalence. Instructions for completing Table III.4
|
G. Methods
Purpose |
Describe a credible and well-specified analytic approach to estimating program effectiveness. This should include a description of the methods you used to account for potential underlying differences between the intervention and comparison groups. |
Instructions and reminders |
Describe the analytic methods you used to answer the primary and secondary research questions. Briefly discuss the analytic method for the primary research questions. For the secondary research questions, note whether the analytic method differs from the analytic method for the primary research questions. These analytic methods should match the approved approach from your analysis plan; if you are making changes from your analysis plan, consult your RETA. For the main impact analysis, briefly summarize (1) the analytic model, including the covariates included; (2) how you handled missing data; and, (3) if applicable, information on sample weights, multiple comparisons, and other items related to study design (for example, clustering correction or covariates for strata). Include equations for estimating impacts in the appendix for transparency, along with any technical details not included in the body of the text. If the study is a QED or an RCT with high attrition at the unit of assignment, we recommend your matched sample is derived using a complete case sample (that is, no imputation of missing data). If you use different methods for your secondary research questions or additional research questions, add a subheading and then describe those methods. Describe any details about data cleaning in the appendix. In addition, if you employed alternate approaches to handling missing data, or if you tested alternate model specifications, include that information in an appendix and reference the appendix in this section. |
Potential sources |
|
Non-text elements |
None |
IV. Implementation evaluation design
Purpose |
Describe the research questions you investigated for the implementation study and how the data collected served to answer those questions. |
Instructions and reminders |
Describe the research questions guiding the implementation evaluation for each aspect of implementation examined (fidelity to the curriculum or program model, dosage of the program, quality of implementation, engagement of participants, and experiences of the comparison group and other context). Then, for each implementation aspect, describe the data that will be used to answer each question, including the sources, and how and when you collected the data. Include research questions related to your core curriculum, as well as the APS topics your program covered. Finally, describe what analyses you conducted. What measures did you construct for the analyses from the data collected? How did you quantify implementation elements? What methods did you use to analyze the data? If detailed descriptions of particular implementation analyses are required, please include this information in the appendix. |
Potential sources |
|
Non-text elements |
Include a table to describe data collected in the appendix, and mention the table in the main body of the report. See Appendix B and Table B.1. |
V. Evaluation findings
A. Implementation evaluation findings
Purpose |
Provide information on the actual experiences of youth in the program and comparison groups. |
Instructions and reminders |
This section should provide information on the program as youth received it (rather than the intended implementation, which you discussed in an earlier section) and the context in which it was delivered. This section should also provide information on the comparison group experience. Write the findings concisely and ground them in numeric findings. (For example, “The program was implemented with fidelity and the program achieved its goals for attendance in this out-of-school program. In all, 95 percent of all program sessions were delivered, and 82 percent of the sample attended at least 75 percent of program sessions.”) Avoid jargon or overly technical terms as much as possible so that a reader without a research background can understand. Use this section to tell the story of implementation, providing context for the impacts and the key lessons learned from implementation. Again, be sure to discuss the implementation findings related to APS content. We encourage the use of subheadings in the text of this section to discuss the findings related to fidelity and dosage, quality of implementation and engagement, and experiences of the comparison group and context. Important: If any unplanned adaptations to implementation occurred during the program, you should describe these adaptations here as part of the findings of the implementation evaluation. |
Potential sources |
|
Non-text elements |
Please refer to the table shells document, which includes 508-compliant table shells for Tables V.1. Instructions for completing Tables V.1
|
Non-text elements (Continued) |
|
B. Impact evaluation findings
Purpose |
Present the impact results for the primary and secondary research questions. |
Instructions and reminders |
Present impacts of the program in tables and then discuss the findings in the text. We recommend that one subsection (and table) shows impacts for primary research questions and a separate subsection (and table) shows impacts for secondary research questions. Make sure each finding answers a given research question. Please present and discuss the findings so that a lay reader (without a research background) can understand them. Avoid jargon or overly technical terms as much as possible. Please present the findings in a metric (for example, difference in proportions) that is easy for readers to interpret. For example, if the outcome is youth reporting sexual activity in the past 3 months, and the estimation method is logistic regression, the estimated coefficients, log odds-ratios, are not easily interpretable. Most statistical software packages will also display the odds-ratio, but that is not much more interpretable: the odds ratio for the treatment variable is the ratio of the odds of a treatment youth reporting sexual activity to the odds of a control youth reporting sexual activity. To produce a more easily interpretable impact estimate, request the mean marginal effects from your software package (using the “margins” command in STATA or using the “%margins” macro in SAS) and report it as the difference in prevalence rates of the outcome across conditions. However, we recommend using ordinary least squares regression as it is the easiest way to get an interpretable result and, as noted earlier, is a valid way to model impacts for binary variables. See this resource for more information. Briefly elaborate on the findings and patterns of findings in this section, but save the broader discussion for the conclusion. (You should, for example, delay tying together implementation and impact findings until the conclusion.)
Include a summary of the similarities and differences in the impact findings across the sensitivity analyses. Include these sensitivity results in Appendix C and Table C.1 or C.2. NOTE: If your analysis plan included additional analyses beyond impact analyses, you can include them in a section V.C.: Additional analyses. |
Potential sources |
None |
Non-text elements |
Please refer to the table shells document, which includes 508-compliant table shells for Tables V.2 and V.3. Instructions for completing Tables V.2 and V.3
|
VI. Conclusion
A. Summary
Purpose |
Summarize the impact and implementation findings.
|
Instructions and reminders |
Restate the main impact and implementation findings and weave the impact and implementation findings together to create a coherent story about how program implementation might have influenced or provide additional context for any observed impacts (or lack thereof). For example, explain how program adherence, youth attendance, or implementation quality might help explain the observed impacts. |
Potential sources |
|
Non-text elements |
None |
B. Limitations
Purpose |
Describe any limitations of the study. |
Instructions and reminders |
Describe the limitations of the study (for example, issues with randomization, study power, or implementation). Discuss how the limitations may influence the interpretation of your findings. For instance, if you had very low attendance in one cohort of youth, that likely meant there was a limited contrast between the actual experiences of the intervention and comparison groups in that cohort (as both groups had similar experiences of receiving none, or very little, of the intervention). If you found no statistically significant findings, this limitation around the contrast could help explain the findings. |
Potential sources |
|
Non-text elements |
None |
C. Discussion
Purpose |
Synthesize the information presented and describe lessons learned.
|
Instructions and reminders |
Present the implications of your evaluation and findings for the broader field. Discuss important lessons learned that explain the impacts or that could help others replicate the program or serve the same target population. For example, if you provided an online intervention, discuss how technology contributed to your evaluation and can be used in the future to address adolescent health education. Also include any areas for future research that you have identified based on this evaluation. |
Potential sources |
|
Non-text elements |
None |
VII. References
Purpose |
Provide the full reference for any work cited in the report. |
Instructions and reminders |
Please use the American Medical Association’s style guide for citing works in the report. This section should include the full reference for any work cited. |
Potential sources |
None |
Non-text elements |
None |
VIII. Appendices
Based on our guidance for the report sections, your report might include the following appendices (though it might not be necessary to include appendices for all of these items, in which case you should relabel your appendices to be sequential):
Please describe the alternate approaches used to clean and analyze the data and present baseline equivalence and impact tables for these sensitivity analyses. In addition, if you are looking at changes over time in a particular measure, and the sample size changes substantially across follow-up periods, you should present a sensitivity analysis that looks at the results for each follow-up using a single sample (in other words, youth who responded to all the surveys). Looking at a single sample with data at each time point will allow you to demonstrate that compositional differences between the samples are not contributing to the observed results. If the results from the sensitivity analyses differ substantially from the main results presented in the report, please provide detailed discussion on which set of results is more appropriate (expanding upon the summary provided in section IV.B). Please use tables to highlight alternate sensitivity results (see Table S.1 in the templates document for a table shell to describe sensitivity analyses for primary research questions and Table S.2 for a table shell for sensitivity analyses for secondary research questions). For each sensitivity analysis, provide a paragraph describing how the result differs from (or replicates the result of) the benchmark approach. Conclude with a brief statement of the differences in magnitude, significance, or both. |
Instructions for completing Tables S.1 and S.2
|
1 For information about Section 508 compliance, please see https://www.hhs.gov/web/section-508/index.html.
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
File Title | Mathematica Standard Report Template |
Author | DPA-Fitts |
File Modified | 0000-00-00 |
File Created | 2023-09-12 |