Cross-Site Evaluation Questions

Project AWARE-TISS Cross-Site Evaluation Questions

The Project AWARE-TISS Cross-Site Evaluation questions (included in the Task Order for the evaluation) were developed to address gaps in knowledge regarding grantee program implementation, training outcomes, youth experience outcomes, and continuity of care.

Project AWARE-TISS Cross-Site Evaluation Questions

AWARE Evaluation Questions

A1. Is AWARE program implemented as intended across the sites?

A1.a. What are the challenges to partnership development, collaboration, and implementation, including training across grantees?

A1.b. What innovative strategies were developed and for which communities and populations?

A2. What are the variations in program implementation across sites and the underlying factors for these variations in implementation?

A2.a. What other resources (implementation science TTA, tool kits) are needed to successfully implement the grant goals?

A2.b. What other funding sources are grantees using to implement the grant?

A2.c. To what extent are grantees implementing the three-tiered approach that is culturally competent, trauma-informed, developmentally appropriate, evidence-based, or evidence-informed?

A3. How do participants of the grant program (school personnel, students, parents, state/local agency, and community partners) describe their experiences in the program?

A3.a. Do such experiences vary by grantee demographics, socioeconomic factors, region, urban vs. rural, existing culture of school personnel regarding evidence-based intervention and existence of minimum workforce?

A4. What are the significant infrastructure plans/changes grantees made in the communities across the sites in improving, expanding, and/or sustaining mental health services for school-aged children when the grants end?

A4.a. How many organizations entered into formal written inter/intra-organizational agreements to improve mental health–related practices/activities that are consistent with the goals of the grant?

A4.b. How many and what type of policy changes were completed because of the grant?

A5. To what extent did the workforce development plan across the grantee sites help increase mental health awareness and literacy?

A5.a. How does training in mental health literacy impact the ability of school personnel to identify and refer youth to mental health services?

A5.b. How many individuals received training in prevention or mental health promotion?

A5.c. What percentage of those who have received training demonstrated improvement between pre- and post-test in knowledge/attitudes/beliefs related to prevention and/or mental health promotion?

A6. To what extent was implementation of the three-tiered public health, pyramid model of intervention successful in school settings?

A6.a. How did it help improve the mental health outcomes for subpopulation (age groups, demographics, urban/rural/ family structure, etc.)?

A6.b. How many individuals were screened for mental health or related interventions?

A6.c. How many individuals were referred to mental health or related services?

A6.d. What is the percentage of individuals receiving mental health or related services after referral (access)?

A6.e. To what extent did participation in a suicide awareness and prevention training affect students’ awareness, knowledge, skills, and self-report behavior?

A7. To what extent did existence of Memorandums of Understanding (MOUs) support clear referral pathways and ensure that school-aged children and youths who need more assistance than brief intervention are referred to and receive necessary school-based and/or community-based mental health services?

A8. To what extent did AWARE grants increase coping skills and resiliency among students across the sites, especially when faced with the challenges of the COVID-19 pandemic?

A8.a. What is the effect of AWARE grants on school safety across grantee sites?

A9. To what extent did AWARE grantees effectively tailor their programs?

A9.a. How do the grantees plan to sustain their programs after federal funding ends?

A9.b. What are some of the barriers and facilitators to sustaining their programs?

A9.c. How are some grantees and their partners developing and implementing a school-based suicide awareness and prevention training policy that is evidence-based and culturally and linguistically appropriate?

A9.d. What evidence-based trainings are provided to students?

A10. What are the overall impacts and/or significant outcomes of this program and how effective was this program across the sites based on the analysis of National Outcomes measures (IPP and other performance data) reported through SPARS and information collected through the review of grantee reports, site visits, and focus groups?

A10.a. To what extent do they describe improvements in resiliency and other school- and individual-level outcomes over the course of their schools’ funding?

A11. How many individuals were trained by AWARE grantees to recognize and intervene in signs of suicidal thoughts and behavior, and how does this vary by age and grade level?

A11.a How do identification and referral volumes vary based on implementation of a suicide awareness and prevention training policy?

A11.b To what extent are grantees implementing suicide awareness and prevention training policy successful in increasing help-seeking reports among students?

TISS Evaluation Questions

T1. How many collaborative partnerships with local/community trauma-informed support and mental health service systems were developed across and engaged across the sites?

T2. What are some of the site-specific challenges and benefits (schools vs. partner agency sites) where mental health services were provided?

T3. Do grantees develop and implement school-based, trauma-informed support and mental health services plans?

T3.a. Do grantees identify barriers to accessing mental healthcare and include a plan to remove those barriers?

T3.b. Do grantees provide evidence-based/informed/best practice trauma-informed supports and mental health services to children, youth, and their families?

T4. To what extent do grantees develop and implement training plans for teachers, teacher assistants, school leaders, specialized instructional support personnel, and mental health professionals to foster safe and stable learning environments and prevent/mitigate the effects of trauma (including through social and emotional learning)?

T5. To what extent do grantees develop and implement family and community engagement plans to increase awareness of the impact of trauma on children and youth, including sharing best practices with law enforcement, regarding trauma-informed care; involving mental health professionals to provide interventions; and longer-term coordinated care within the community for children, youth, and their families who have experienced trauma?

T6. Do grantees make sustainability plans that describe how the project activities can be continued/sustained?

T7. Do grantees establish local interagency agreements among LEAs, agencies responsible for early childhood education programs, Head Start agencies, juvenile justice authorities, mental health agencies, child welfare agencies, and other relevant agencies in the community?

T8. What are some of the allowable activities grantees implemented and grantee evaluation findings including challenges and benefits of such?

T9. What are some of the innovative activities and evidence-based practices grantees implemented?

T9.a. Is the TISS program implemented as intended across the sites?

T9.b. What are the variations in program implementation and the underlying factors influencing the variations?

T9.c. What collaborative efforts have contributed to the improvement in identification, referral, early intervention, treatment, and support services for students?

T9.d. How do the grantees engage families and communities to increase the awareness of trauma impact on children and youth?

T10. What are the overall impacts and/or significant outcomes of this program and how effective was this program across the sites based on the analysis of National Outcomes measures (IPP and other performance data) reported through SPARS and information collected through the review of grantee reports, site visits, and focus groups?

T10.a. To what extent do they describe improvements in resiliency and other school- and individual-level outcomes over the course of their schools’ funding?

T10b. To what extent do students identified by TISS grantees report improvements in functioning, social connectedness, and other mental health outcomes following engagement in mental health services? How does this vary by grantee (e.g., activities, partnerships) and participant (e.g., race, ethnicity, sexual identity, gender identity) characteristics?

T11. To what extent does TISS improve school capacity to identify, refer, and provide services to students in need of trauma support or behavioral health services?

T11.a. To what extent did TISS reflect the best practices for trauma-informed identification, referral, and support developed under section 7132 for the Interagency Task Force for Trauma-Informed Care?

T11.b. How many individuals were identified as in need of trauma support or behavioral health services?

T11.c. How many individuals were referred to evidence-based and culturally relevant trauma support services and mental healthcare?

T11.d. What is the percentage of individuals receiving mental health or related services after referral?

Cross-Program Evaluation Questions

AT1. How do the factors contributing to effective collaboration translate to effective systems changes?

AT2. What is the type and extent of systems change?

AT2.a. Is there a change in quality of mental health service delivery for school-aged children and youth and increase in the number of quality programs/interventions accessible and available?

AT3. Is there a change in the comprehensiveness of services provided in schools as a result of system change?

AT3.a. Do grantees ensure that there is a mix of services available to meet the needs of subpopulations and under-resourced communities?

AT4. Does the effectiveness of identification and referral systems vary based on grantee implementation of practices meant to address behavioral health disparities?

AT4.a. How does the proportion of identified students who receive follow-up services vary based on the characteristics of the youth identified (e.g., race, ethnicity,

sexual identity, gender identity) or identifying staff (e.g., role type, training)?

AT4.b. To what extent is the proportion of students receiving services after referral reflective of overall school demographic profile?

AT5. Do specific grantee approaches have differential impacts on reducing disparities in mental health outcomes over time?