Appendix C: Impact CONSORT Instructions

CONSORT Instructions

The information collected will be used for internal purposes to (1) assess the sample enrollment and compare actual sample sizes to the target sample sizes on which power calculations were based, and (2) assess the likelihood that the final analytic sample for key follow-up time periods might exceed the thresholds of overall or differential attrition defined by the HHS evidence standards.

Which CONSORT diagram template(s) should you use?

There are two separate sets of CONSORT diagram templates for PREIS grantees to select from. The templates were designed to accommodate differences in the order of assignment and study consent. There are templates for (1) studies that obtain consent from participants prior to assignment, and (2) studies that obtain consent from participants after assignment. Please select the set of templates that best matches the order of your study’s activities.

Each set of CONSORT diagram templates contains two diagrams, one for clusters and one for youth. If your study involves a cluster-level design (in which clusters such as schools, classrooms, or groups of youth are assigned to a study condition), please complete both diagrams. If your study involves an individual-level design (in which each individual is assigned to a study condition), please complete only the CONSORT diagram for youth.

To complete the CONSORT diagram(s), use the most recent evaluation data available related to enrollment and data collection activities.

CONSORT diagrams for clusters (if applicable)

All sample sizes in the CONSORT diagram for clusters refer to the numbers of clusters.

1. In the top box, please include a brief paragraph (2-4 sentences) describing how clusters are defined; what makes a cluster eligible for the evaluation; the number of clusters screened; the number of clusters determined to be eligible; and whether and how any clusters were prioritized for inclusion in the study sample.

2. In the “Key dates” box, please indicate the date through which sample was enrolled and the date through which survey data were collected and are reflected on this diagram.

   1. Example 1: A study enrolls youth at the beginning of each school year in September, and collects baseline data in October, posttest in April, and first follow-up data in July). The grantee completes the CONSORT diagram in July 2020, using enrollment data through the prior September 2019 and data collected through April 2020. The date for enrollment data would be September 2019, while the date for survey data would be April 2020.

   2. Example 2: A study enrolls individuals on a rolling basis, and collects baseline data at the time of enrollment, posttest data at the end of the last program session, and first follow-up data three months after the program ends. The grantee completes the CONSORT diagram in July 2020, using enrollment and survey data through May 2020, as this was the latest data processed. The date for enrollment data and the date for survey data would both be May 2020.

3. In the “Clusters Randomized” box, please indicate the number of clusters (randomly) assigned in total to the study groups (i.e., treatment or comparison), and the start and end dates of cluster assignment.

4. In the box starting with “Did not pass screening criteria,” please list the reasons that screened clusters did not participate in the study. The categories should be mutually exclusive and the sample sizes should add up to the number of clusters that were excluded from the study.

5. In the boxes “Assigned to Treatment” and “Assigned to Comparison,” please indicate the number of clusters assigned to the treatment and comparison groups, respectively.

6. For the remaining boxes, please indicate the number of clusters still participating in the study, by study condition, at each data collection time point. A cluster is defined as “participating” if at least one individual in the cluster completed the data collection effort.

   1. Grantees should note all reasons clusters dropped out and the number of clusters each reason applies to. All non-response categories should be mutually exclusive and have sample sizes that add up to the total number of clusters that dropped out of the study.

   2. In the two boxes in the bottom row, the “primary analysis sample” refers to the number of clusters in the main sample that will be used to answer the primary research questions in the final report. You will define your primary analysis sample using one of the follow-up survey samples reported earlier in the CONSORT. In addition to accounting for non-response on the follow-up survey, you will then also account for sample loss due to missing data and any techniques used to establish an equivalent sample. Document the number of clusters dropped from the analytic sample due to each issue (e.g. missing follow-up data, missing baseline data, item non-response, or matching techniques), and indicate the total number of clusters remaining in the analytic sample.

CONSORT diagrams for Youth

All sample sizes in CONSORT Diagram for Youth refer to the numbers of individuals.

1. In the top box, please include a brief paragraph (3-4 sentences) describing what makes a youth eligible for the evaluation; the number of youth screened and determined to be eligible; and the process for selecting study participants among those who were eligible (if applicable).

2. In the “Key dates” box, please indicate the date through which sample was enrolled and the date through which survey data were collected and are reflected on this diagram. (See examples above.)

3. In the “Randomized” box, please indicate the number of youth (randomly) assigned in total to the study groups and the start and end dates of assignment.

4. In the box starting with “Did not receive parental consent,” please list the reasons that screened youth did not participate in the study. The categories should be mutually exclusive and the sample sizes should add up to the number of youth who were screened and either found to be ineligible or who did not agree to participate in the study.

5. In the boxes “Assigned to Treatment” and “Assigned to Comparison,” please indicate the number of youth assigned to the treatment and comparison groups, respectively.

6. Similarly, in the boxes “Agreed to be in study,” please indicate the number of youth who consented to the study, by treatment and comparison groups, respectively.

7. For the following boxes, please indicate the number of youth that provided data, by study condition, at each data collection time point (baseline and subsequent follow-ups), the start and end dates of data collection, and the reasons for non-response.

   1. In the “Eligible for [X (e.g., immediate post)] follow-up” field, please indicate the number of youth who reached the corresponding data collection event. While the study is in progress, some individuals may not have had the opportunity to contribute data for a particular data collection event. For example, individuals who have not completed the program would not have had the opportunity to contribute any follow-up data. Therefore, for any CONSORT diagram completed prior to the end of all data collection activities, it is important to document the number of youth who were eligible as of the key date for survey data indicated above, in addition to providing the number of youth who actually did provide data.

      1. In Example 1 above, in July 2020, youth who enrolled in September 2019 would have provided posttest data in April 2020, but would not have completed the first follow-up. Therefore, the youth enrolled in September 2019 should be excluded from the number of youth eligible for the first follow-up.

      2. In Example 2 above, in July 2020, youth who enrolled in May 2020 would have provided baseline data, but would not have provided posttest or first follow-up data in May 2020. Therefore, the youth enrolled in May 2020 should be excluded from the number of youth eligible for both the posttest and first follow-up.

   2. If the evaluation uses a cluster-level design, then the number of youth in each condition should reflect the number of youth only in participating clusters at that time point. Youth in clusters that have dropped entirely from the study should be excluded from these counts.

   3. Grantees should note all reasons for non-response, and the number of youth each reason applies to.

   4. In the two boxes in the bottom row, the “primary analysis sample” refers to the main sample that will be used to answer the primary research questions in the final report. You will define your primary analysis sample using one of the follow-up survey samples reported earlier in the CONSORT. In addition to accounting for non-response on the follow-up survey, you will then also account for sample loss due to missing data and any techniques used to establish an equivalent sample. Document the number of youth dropped from the analytic sample due to each issue (e.g. missing follow-up data, missing baseline data, item non-response, or matching techniques), and indicate the total number of youth remaining in the analytic sample.

8. Finally, in the boxes in the middle, provide the program start and end dates. If the program was implemented to youth in cohorts, provide the dates by cohort.