Supporting Statement A

Project AWARE-TISS Cross-Site Evaluation

Project AWARE and TISS Programs

Supporting Statement

Check off which applies:

☒New

☐Revision

☐Reinstatement with Change

☐Reinstatement without Change

☐Extension

☐Emergency

☐Existing

1. Justification

The Substance Abuse and Mental Health Services Administration’s (SAMHSA’s) Division of Trauma and Disaster Behavioral Health (DTDBH) is requesting Office of Management and Budget (OMB) approval for a national cross-site evaluation of process, outcomes, and associated impact for the Advancing Wellness and Resilience in Education (hereinafter referred to as Project AWARE) and Trauma-Informed Services in Schools (TISS) programs. This evaluation reflects SAMHSA’s desire to better understand how each program is implemented, the extent to which they facilitate collaboration between education agencies and mental health systems, and how each program contributes to referral to mental healthcare and improved outcomes for youth.

1. Circumstances of Information Collection

Our nation’s children and youth are facing an unprecedented mental health crisis. Even before the COVID-19 pandemic, rates of depression, anxiety, and suicidal thoughts among youth were on the rise (American Academy of Pediatrics, 2021; Bitsko et al., 2022). Recent findings indicate that 20% of all children have an identified mental health condition each year, while 40% of all children will meet criteria by age 18 (Bitsko et al., 2022). In addition, children living in poverty and in racial and ethnic minority populations fare worse than their peers with respect to access to care, identifiable risk factors, and prevalence of certain mental health conditions. Despite high rates of mental health conditions among children and youth, Bitsko et al. (2022) also report low rates of treatment: about 11.4% annually for White, 9.8% for Black, and 8.7% for Hispanic/Latino children.

Beginning in April 2020, the proportion of children’s mental health–related ED visits among all pediatric ED visits increased and remained elevated over the next 6 months. Compared to 2019, the proportion of mental health–related visits for children aged 5–11 and 12–17 years increased approximately 24% and 31%, respectively (Leeb et al., 2020). Moreover, an analysis of private healthcare claims data indicates that use of services among youths related to mental health conditions, intentional self-harm, overdoses, and substance use disorders also increased in the interim from the start of the pandemic, underscoring the toll the pandemic exacted on youth behavioral health (FAIR Health, 2021). Specifically, the pandemic added to challenges already faced by America’s children and youth. It disrupted in-person schooling; positive relationships and routines, including in-person social opportunities with peers and mentors; access to healthcare, social services, food, and housing; and the health of caregivers (Wachino et al., 2021). One study estimated that more than 140,000 children in this country lost a primary and/or secondary caregiver, with youth of color disproportionately impacted (Hillis et al., 2021). In addition, the pandemic presented new barriers to youth access to mental healthcare, with providers and the mental health system as a whole operating beyond capacity (Wachino et al., 2021). Well over a year into the pandemic, more than half of parents and caregivers expressed concern over their children’s mental well-being (American Psychiatric Association, 2021).

National leaders, experts, researchers, and youth-serving practitioners have responded by bringing attention and committing resources to understand and address these issues, most recently with the U.S. Surgeon General’s Advisory on Protecting Youth Mental Health (Office of the Surgeon General, 2021). The advisory highlights the urgent need to address the nation’s current youth mental health crisis and outlines a wide range of causes for the crisis and related potential solutions and recommendations. These include an expansion of school-based mental health services as a critical strategy to address the child and youth mental health crisis. Administered by CMHS, AWARE and TISS are designed to address the youth mental health crisis by responding to national calls for a stronger continuum of supports to meet student mental health needs, including evidence-based prevention practices and trauma-informed mental healthcare, as well as resources, technical assistance (TA), and collaboration through mental health programs (Office of the Surgeon General, 2021). Both Project AWARE and TISS programs reflect an understanding of the critical role that schools play in ensuring that behavioral health problems are identified early so that children and youth can grow and thrive.

AWARE was first authorized in 2014 as part of President Obama’s Now is the Time Initiative (NITT), which focused on improving school safety not only by enhancing youth physical security and ensuring preparation to respond to emergencies such as mass shootings, but also by creating safer and more nurturing school climates that help prevent school violence. Informed by the growing evidence base related to developing comprehensive school mental health systems and refined since its inception in 2014, AWARE is fundamentally designed to address the impact of the pandemic on America’s children and youth. Going forward, the purpose of AWARE is to develop a sustainable infrastructure for school-based mental health programs and services with a focus on children and youth in grades K-12. Grant recipients will build a collaborative partnership that includes the State Education Agency (SEA), Local Education Agencies (LEAs), Tribal Education Agencies (TEAs), the State Mental Health Agency (SMHA), community-based providers of behavioral healthcare services, school personnel, community organizations, families, and school-aged youth. Continuing to emphasize a public health model, this partnership will focus on mental health–related promotion, awareness, prevention, intervention, and resilience activities to ensure that students have access and are connected to appropriate and effective behavioral health services. SAMHSA expects that this program will promote the healthy social and emotional development of school-aged youth and prevent youth violence in schools.

The purpose of the TISS program, also a key aspect of the President Biden’s current national strategy on mental health (U.S. White House Briefing Room, 2022), is to increase student access to evidence-based and culturally relevant trauma support services and mental healthcare by developing innovative initiatives, activities, and programs to link local school systems with local trauma-informed support and mental health systems, including those under the Indian Health Service (IHS). This program further enhances and improves trauma-informed support and mental health services for children and youth—in part, with recognition that young people have been especially impacted by the pandemic.

During Fiscal Years (FYs) 2021 and 2022, SAMHSA will have funded 32 AWARE grants and 15 TISS grants that will be a part of the national cross-site evaluation. As part of their Notice of Funding Opportunity (NOFO), Project AWARE and TISS grantees are required to participate in the national cross-site evaluation initiative. This may include cooperation with site-visits, interviews with key staff, and assistance in organizing local stakeholder focus groups, sharing SAMHSA’s Performance Accountability and Reporting System (SPARS) data, final program performance reports, and any other OMB approved data collection within the scope of the national evaluation. The Project AWARE-TISS Cross-Site Evaluation serves as a primary mechanism through which the AWARE and TISS Programs will continue to be understood, improved, and sustained. A government contractor (ICF) will coordinate data collection for the evaluation and provide support for its local-level implementation. In this partnership, ICF will also provide training and TA regarding data collection and research design for the evaluation. In addition, ICF directly collects data, receives data from grantee data collection efforts, monitors data quality, and provides feedback to grantees. Data gathered through the Project AWARE-TISS Cross-Site Evaluation will continue to be utilized for both grantee-specific and national assessments of the program.

2. Purpose and Use of Information Collected

The AWARE-TISS Cross-Site Evaluation will provide SAMHSA with critical information needed to better understand and document the effects of the AWARE and TISS programs together and separately, including the extent to which these programs improve access to mental health services for school-aged youth. The information gathered will be used by SAMHSA, their grantees, and other key partners in helping communities and decision-makers at all levels of government improve the youth mental health crisis by strengthening the evidence base for school-based mental health programs.

The focus of the AWARE-TISS Cross-site Evaluation on assessing the implementation, outcomes, and impacts of the Project AWARE and TISS Programs will contribute immensely to building a stronger continuum of support to meet student mental health needs. Without this evaluation, Federal and local officials will not be able to determine program effectiveness on identification, referral, and provision of services to youth and students identified as at risk; and whether Project AWARE and TISS grantee programs are meeting their goals. SAMHSA will use the data collected to provide objective measures of progress toward meeting key performance indicators put forward in its annual performance plans as required by law under GPRA.

The Project AWARE-TISS Cross-Site Evaluation includes four evaluation components to provide a robust understanding of the implementation, outcomes, and impacts of the AWARE and TISS Programs: (1) a process evaluation; (2) an outcome evaluation; (3) program-specific sub-studies; and (4) cross-program analyses. A behavioral health equity and cultural equity lens will be applied to each area of evaluation to ensure a culturally specific understanding of evaluation findings.

The Process Evaluation will contain two studies (Implementation and Sustainability Study and Systems Change Study) that examine strategies common to both programs related to program implementation facilitators and barriers, workforce development, and grantees’ plans to sustain critical program components beyond their grant period. This assessment of common elements will provide a means to compare the implementation strategies that are successful across both AWARE and TISS grantees and identify successes and challenges in changing systems, policies, service provision, and school climate; increasing behavioral health equity in access and service delivery; and increasing social and emotional development and well-being in school-aged children and youth. The Process Evaluation will also address implementation of program-specific components.

For AWARE, the evaluation will document how the grantees implement the three-tiered public health model in schools and the referral pathways to increase access to mental health related promotion, prevention, and intervention. The evaluation will assess the grantee collaborative efforts and grantee activities intended to increase workforce capacity to identify the signs and symptoms of mental illness and ability to refer them to appropriate services promptly.

For TISS, the Process Evaluation will focus on examining what innovative strategies the grantees use to increase access to trauma informed services for school-aged youth and how the collaborative efforts of grantees and their partners develop/improve a school-based system for identification, referral, early intervention, treatment, and supportive services. Additionally, the Process Evaluation will assess the implementation of training to improve school capacity to address trauma support needs and engagement of families and communities to increase awareness of the effects of trauma on children and youth.

The Outcome Evaluation comprised of two studies (Identification and Referral Study and Youth Resiliency and Outcomes Study) will provide critical information about the effectiveness of the AWARE and TISS programs in establishing and enhancing school-based mental health supports for students. Pairing evidence from the Outcome Evaluation with insights from the Process Evaluation also facilitates understanding of the specific program and contextual factors that are most closely linked with outcomes in two primary areas of focus: identification and referral infrastructure and youth resiliency and outcomes.

Program specific sub-studies, inclusive of two TISS case studies and an AWARE Suicide Awareness and Prevention Sub-Study, will be conducted to provide more extensive contextual and implementation information related to the two programs.

Finally, in addition to assessing the process and outcomes of each of the AWARE and TISS programs, two cross-program analyses (Cross-Program Impact Analysis and Behavioral Health Equity Cross-Study Analysis) will be conducted that examine the impact of the both programs on the establishment and enhancement of school-based mental health supports for students and the relationships of program and contextual factors with outcomes.

The proposed multimethod approach considers allowable and required activities, variation in the partnerships and provider networks/infrastructure, program settings, populations being served, the range of program implementation plans and goals, existing data systems, and grantee infrastructure that support implementation and evaluation participation. In addition, the design considers the stage of implementation of currently funded grantees to seamlessly integrate cohorts appropriately into the evaluation studies.

2. Use of Technology and Description of Data Collection Methods

Every effort has been made to limit the burden on individual respondents who participate in the AWARE-TISS Cross-Site Evaluation through the use of technology. Quantitative data collection instruments will be administered via a web-based portal referred to as the AWARE-TISS Online Data System (ATODS). The ATODS will serve as a (1) data entry tool for grantee program and evaluation staff to enter information and (2) data collection tool for administering web surveys to respondents. All data obtained through direct entry by grant program staff or through web surveys will be stored in the central repository in the SAMHSA cloud environment to reduce evaluation burden on grantees and to allow ease of access to data for program personnel and evaluation team members. All data collected will be stored in the central data repository, allowing for the analysis and summary of information within and across surveys.

The web-based ATODS will support data collection, management, and dissemination activities associated with the AWARE-TISS Cross-Site Evaluation, including communication between grantees and the evaluation team, secure data transmission and storage, data quality monitoring that triggers corrective action when necessary, and updates around evaluation activities and performance. Through the ATODS, grantees can:

• Manage their own user profile and add subordinate users with same or limited permission

• Access web-based data collection forms, surveys, and data import tools, designed responsively to support varying screen sizes across a wide spectrum of devices (e.g., desktop, laptop, tablet, phone)

• Track progress and completeness at the instrument level and by categories or major sections within each individual data collection tool

• Respond to real-time validations during data entry or upload, fixing errors and inconsistencies prior to submission and reducing the effort of responding to post-submission data questions. Front-end validations will prevent invalid or incomplete data from being saved into the database. These types of validations can be warnings that allow a user to proceed or hard errors that must be corrected before proceeding. Using validations and data reports, the system will allow for quick identification of inaccuracies and anomalies in the data and allow for corrective action.

• Access and download grantee-specific data submission metrics and datasets

• Access general and targeted evaluation-related documents and tools (e.g., data dictionaries, codebooks, user manuals, links to websites)

To maintain privacy, the secure ATODS offers six levels of password-protected access to site-specific and aggregate data as described in Exhibit 1. Only users with administrative privileges (evaluation management, evaluation team, and grantee site administrators) will have the security to access the raw data. To protect from potential misuse of those data (inadvertent identification of respondents as a function of their unique demographic information), the following measures will be in place: (1) access to raw datasets will be restricted to designated individual(s) and (2) the grantee site administrator will sign a data use agreement. Within the context of protecting from inadvertent identification, this agreement will stipulate who, how, and under what circumstances the raw data can be analyzed/reported. For example, the AWARE-TISS Cross-Site Evaluation team will obtain an agreement from each grantee site administrator not to report categories in which less than 10 cases exist and to stipulate who will have access to raw data. Further, the agreement will indicate that no attempt, through complex analysis using outside information, will be made to ascertain the identity of individuals from the datasets.

Exhibit 1. ATODS User Security Levels

Security Level	ATODS Privileges
Evaluation Management Administrator	Capability to view number of instruments completed & submitted (all grantees) and to download site and aggregate datasets Access site-specific data from all grantees and available aggregate reports Capability to add, modify, and remove users for all grantees
Evaluation Team Member Administrator	Capability to view number of instruments completed & submitted (all grantees) Capability to download site and aggregate datasets Access site-specific data from all grantees and available aggregate reports
Grantee Site Administrator	Capability to view number of instruments completed & submitted and to download site-specific datasets Capability to enter data via upload or web survey Access grantee-specific data and reports and available aggregate reports Capability to add, modify, and remove users for site
Grantee Site User	No access to view the number of instruments completed and submitted No access to datasets Access to aggregate reports only Capability to enter data via upload or web survey
Grantee Contact User	Capability to enter data into the web-based system only No other privileges (for data collectors and survey respondents)
SAMHSA, its Consultants, & Partners	Capability to view number of instruments completed & submitted (all grantees) Access site-specific data from all grantees and available aggregate reports

The newly designed ATODS is currently under development and the System of Records Notice (SORN) and HHS Privacy Impact Assessment (PIA) have yet to be created.

Approval is being requested for 14 new data collection instruments that comprise the AWARE-TISS Cross-Site Evaluation (Exhibit 2). A full list of accompanying attachments is located at the end of Supporting Statement B.

Exhibit 2. Project AWARE-TISS Cross-Site Evaluation Instrument List with Acronyms and Status

Attachment	Acronym	Name	Instrument Status
A	IS	Implementation Survey	New
B	IKII	Implementation Key Informant Interview Guide	New
C.1	YFFG-Y	Youth and Family Focus Group Moderator Guide -Youth	New
C.2	YFFG-F	Youth and Family Focus Group Moderator Guide - Family	New
D	CPS	Collaboration and Partnership Survey	New
E	TSF	Training Summary Form	New
F	PFF	Participant Feedback Form	New
G	APPTS	AWARE Pre-Post Training Survey	New
H	TPPTS	TISS Pre-Post Training Survey	New
I	WFS	Workforce Follow-up Survey	New
J	PCSS	Parent Climate and Safety Survey	New
K	STCSS	Student Climate and Safety Survey	New
L	SSCSS	School Staff Climate and Safety Survey	New
M	SIRF	Student Identification and Referral Form	New

AWARE-TISS Cross-Site Evaluation Instruments

IS: The web-based IS will be conducted annually for three years to assess program implementation from a state, local, and community perspective. This survey is adapted based on the IS administered as part of the SAMHSA Safe Schools/Healthy Students Evaluation and includes questions on protocols, policies, and structures present as part of schools’ AWARE and TISS implementation processes; school/community integration; and barriers and facilitators to implementation. In addition, the survey incorporates the Program Assessment Sustainability Tool (Luke et al., 2014), a 40-item assessment that has been tested for reliability and focuses on program sustainability capacity across eight domains. The survey also includes questions to gather program-specific information—for example, implementation of the pyramid model in the case of AWARE grantees and details on trauma-informed services in the case of TISS grantees. Before and during the administration period, invitations and reminders will be sent via email to designated participants. Respondents will be asked to review a virtual informed consent form just before completing the survey. The IS will be completed by AWARE SEA, LEA/TEA, SMHA project coordinators (n=128); TISS project coordinators (n=15) and other program or partner staff (n=15); the survey will be conducted annually over three years (n=158).

IKII : Supplementing IS findings, ICF will conduct IKIIs to obtain in-depth information about AWARE and TISS program implementation and sustainability based on the perspectives of grantee program staff and local mental health system partner staff. In each year of the data collection period, ICF will conduct 6 interviews at each grantee site with key representatives of the grantee’s collaborative partnership group. ICF will use a semi-structured interview guide including open-ended questions focused on partnership development, coordination, and shared decision-making; perspectives on implementation including challenges, strategies, and successes; contextual, systems, or other factors that affect implementation; and approaches to planning for program sustainability. Technical Assistance Liaisons (TALs) will conduct the interviews in person during TTA site visits when possible or virtually as needed. The team will draw on extensive experience conducting virtual interviews using Zoom, MS Teams, and other virtual platforms. To recruit respondents for these interviews, ICF will provide guidance to grantees on the types of participants we would like to interview to support development of a purposeful sample of individuals who are especially knowledgeable about the implementation process in each community (Patton, 1990). Grantees will coordinate recruitment of respondents. Once respondents have been identified, TALs will then reach out via email and phone to schedule the interviews, to be conducted in person during the site visits or remotely if needed. ICF will provide an informed consent form via email in advance of the interview and obtain verbal consent immediately prior to each interview. Each interview will last approximately 45-60 minutes. Interviewers will review the consent form and answer any questions before beginning the interview. With the participant’s permission, interviews will be recorded and transcribed to support fact checking and to maintain the reliability and validity of the data.

YFFG-Y and YFFG-F: The YFFG-Y and YFFG-F will be used to conduct focus groups with youth who attend schools implementing the AWARE or TISS programs or their parent(s)/family representatives. The moderator guides will be semi-structured and include open-ended questions to understand experiences and perspectives related to school climate, positive supports, youth or parent engagement, student resiliency and coping skills, awareness of school-based programs or resources to promote mental health literacy and meet mental health needs, mental health resource availability, and satisfaction with the program. Youth and family focus groups will be conducted annually and will include youth or parents representing 12 AWARE grantees and 6 TISS grantees per year, such that at least one focus group per grantee will be conducted during the evaluation. Each focus group will include a purposive sample of up to 10 youth aged 14-18 (or older if appropriate) or parents/family representatives. We will consider best practices related to conducting focus groups with youth such as avoiding age differences of more than 2 years among youth in the same focus group (Adler, Salanterä, and Zumstein-Shaha, 2019). Focus groups will be conducted during TTA site visits or virtually as requested. Each focus group will last 60-90 minutes. Adult participants will be provided with a copy of the informed consent form prior to participating in the focus group. In addition, prior to beginning a focus group discussion, the moderator will review the consent form, answer any questions, and confirm consent. Parental consent will be required and obtained for all youth prior to participation. Youth will also be asked to provide verbal assent to participate at the time of the focus group. Youth who do not provide assent will be excused from participation. With participants’ permission, ICF will record the focus groups to ensure an accurate record and facilitate transcription.

CPS: CPS will be conducted annually with a sample of state and local entities identified as partners by all AWARE and TISS grantees. Respondents will include AWARE SEA, LEA/TEA, SMHA project coordinators (n=128), local community mental health provider agency staff (n=32), AWARE school administrators (n=32), TISS project coordinators (n=15), TISS local community mental health provider agency staff (n=15), and TISS school administrators (n=15). This 43-item survey is adapted from the State and District Collaboration Matrix used for a previously conducted SAMHSA multi-site evaluation of the Safe Schools Healthy Students State Planning Program and includes questions on communication, working relationships, leadership, role-expectations, resources, and partner engagement/commitment. Respondents will also be asked whether their organization currently has a formal, signed memorandum of agreement with the grantee and what changes to policy and infrastructure have been made in the past year.

TSF: The TSF will be used annually by AWARE and TISS grantees to document training and educational seminars. It will include training dates, length of time of training (in hours), topic of the training, training objectives, format of training delivery (in-person, webinar, online asynchronous, etc.), intended audience, and number of training participants. It is estimated that grantees will conduct up to 10 trainings annually for different groups (e.g., teachers, mental health professionals, instructional support personnel).

PFF: The PFF will be a web-based form distributed annually to up to 75 training participants per AWARE grantee and up to 25 training participants per TISS grantee after training events to gather participants’ assessments of the training experience and perceived feasibility of using the information. This 14-item survey will include the items from the baseline Impact of Training and Technical Assistance (IOTTA) tool (Walker & Bruns, n.d.), which assesses stakeholder perceptions of immediate and longer-term benefits in areas that research has linked to effective implementation and practice change.

APPTS and TPPS: The APPTS and TPPTS are pre- and post-surveys that will be completed before and after grantee training events. The APPTS will be completed by up to 125 AWARE training participants per AWARE grantee annually and includes 51 items to assess mental health literacy, attitudes, beliefs (including stigma), and self-efficacy to provide support and referrals to youth experiencing mental health symptoms using questions drawn from the Mental Health Beliefs and Literacy Scale (Anthony et al., 2015). While this survey was designed to evaluate the Mental Health First Aid training, the questions are applicable to the knowledge, attitudes, and beliefs related to identifying students in need of mental health services and referring them for mental health services (e.g., “Currently, I am confident that I can assist a person who may be dealing with a mental health problem(s) to seek professional help.”)

The TPPTS will be completed by up to 50 TISS training participants per TISS grantee annually. The TPPTS was developed based on the Primary Early Childhood Educators Trauma-Informed Care Survey for Knowledge, Confidence, and Relationship Building (PECE-TICKCR, Bilbrey et al., 2022). The 22 items in this survey assess respondents’ knowledge of and self-efficacy to use trauma-informed strategies in their work, with items like “I can take steps to support a student who has been identified as experiencing a traumatic event.”

WFS: The WFS will measure behavioral changes and longer-term impact on systems and communities which will be administered to approximately 50% of AWARE and TISS training participants that also complete the APPTS or TPPTS. The WFS will be completed 3 and 12 months after training events. The 29-item WFS will include AWARE and TISS-specific items and the IOTTA measure (Walker & Bruns, n. d.) to assess barriers and supports to training use. Questions from the Mental Health Beliefs and Literacy Scale (Anthony et al., 2015) will be used to assess the extent to which participants identified students in need of mental health services and referred them to those services following the training.

STCSS, SSCSS, and PCSS: The STCSS, SSCSS, and PCSS are adapted from the U.S. Department of Education’s School Climate Survey (EDSCLS) to assess school climate and safety (https://safesupportivelearning.ed.gov/edscls) as part of the Project AWARE-TISS Cross-Site Evaluation. EDSCLS is a free, open-source school climate survey available or any local or state education agency to use to measure climate and includes surveys for students (middle and high school), school staff, and parents. The EDSCLS includes 13 scales for school staff and 12 scales for students.¹ To reduce survey burden for respondents, the scales for topics most relevant to AWARE and TISS will be included. In addition to the EDSCLS, 5 additional items are included on the STCSS from the California Healthy Kids Survey (California Department of Education, n.d.) that ask about students’ perception of availability of resources for mental health and whether they received a referral or accessed mental health services. Five additional items from the California School Staff Survey (California Department of Education, n. d.) that directly ask about trauma-informed practices for the SSCSS. To address school racial climate, several items from the School Climate for Diversity Scale (Byrd, 2017) were incorporated into the STCSS, and items were added to SSCSS focused on respect for diversity (Mattison and Aber, 2007; Weinstein et al., 2003). The surveys will be completed by students attending grantee LEA schools and parents (n=6 students and n=6 parents of students per grantee), school personnel (n=6 per grantee), and LEA staff (n=4 per grantee) in year one and in year three. Parental consent will be required and obtained prior to participation for students under 18 years of age.

SIRF: Establishing identification and referral systems, including coordination with support service providers equipped to meet the needs of youth, is a core component of AWARE and TISS grant requirements. The SIRF is a web-based form that details how youth were identified as in need of mental health, substance use, or trauma-specific support services because of an AWARE or TISS program, whether and to which services youth were referred, and resulting services received. Grantee staff will submit SIRF data for up to 100 youth annually identified through an AWARE or TISS program (e.g., through AWARE or TISS program staff, an AWARE- or TISS-trained individual within the school, or through an AWARE or TISS-sponsored screening). Information about the initial identification, including the location and pathway to identification (e.g., individual, screening tool, staff role), will be obtained, along with information about referrals and support services received following identification. The form also includes deidentified demographic information about the youth receiving the identification, referral, and follow-up care. SIRF data can be extracted from case records of school-based care coordinators or mental health providers, or other existing data sources, including any school staff, support service provider, and family members who make a mental health, substance use, or trauma-related identification and referral. Grantees without access to an existing tracking system for referrals should contact their Technical Assistance Liaison (TAL), and SAMHSA Government Project Officer (GPO) to discuss approaches for obtaining data. All SIRF data will be entered on an ongoing basis. The SIRF is intended to be completed by grantee program staff as part of a record review for each youth identified and referred to support services. ICF TALs, with the support of SAMHSA GPOs, will assist grantees in developing an approach to obtain this data that is specific to their local context. It is not anticipated that SIRF data collection will be completed through a self-report format, or otherwise require direct consumer involvement. ICF will make a printable PDF version of the SIRF form available to grantees who prefer to complete the form on paper before entering data into the AWARE-TISS online data system. ICF TALs will support grantees in creating and implementing a procedure to ensure confidential and complete collection of SIRF data.

2. Efforts to Identify Duplication

The AWARE-TISS Cross-Site evaluation team, in developing the data collection activities, consulted existing literature to avoid duplication in data collection activities and the use of similar information. In addition, the evaluation team conducted a comprehensive review of Project AWARE and TISS grant applications, including assessment of grantee program evaluation plans and existing measures when available. ICF also worked closely with our Evaluation Advisory Panel (EAP) that consists of eight experts with a range of knowledge in upstream prevention, childhood trauma and trauma-informed care, adverse childhood experiences, school-based programs, mental well-being and resilience in American Indian and Alaskan Native (AI/AN) and multicultural communities and vulnerable populations.

2. Impact on Small Businesses or Other Small Entities

Some data collection activities may involve individuals from public agencies that provide a wide array of services such as mental health, juvenile justice, education, and child welfare. While most data will be collected from public agencies, it is possible that small businesses will participate in grantee-sponsored AWARE or TISS related trainings and have individuals who serve as training participants included in the Systems Change Study. We anticipate that any data collection efforts will not have a significant impact on these small entities.

2. Consequences if Information Collected Less Frequently

The rigor of the Project AWARE-TISS Cross-Site Evaluation design and its ability to answer the primary evaluation questions is dependent on the frequency of the data collected. Additionally, because the cross-site evaluation is aligned with the key elements of the Project AWARE and TISS Programs, the frequency with which data collection activities are administered is critical to SAMHSA’s overall assessment of the programs. Exhibit 3 describes the consequences if data are collected less frequently.

Exhibit 3. Data Collection Activities and Consequences If Information Collected Less Frequently

Activity	Rationale
IS	The IS assesses program implementation annually from a state, local, and community perspective including the barriers and facilitators to implementation, school/community integration, and sustainability. This survey focuses on various specific areas of interest including policy implementation, pyramid models, referral systems/pathways, innovative strategies, sustainability plans, and suicide awareness or prevention policies. If these data are collected less frequently, we will lose the ability to track AWARE and TISS program implementation over time.
IKII	The IKIIs collect in-depth information annually about AWARE and TISS program implementation and sustainability from grantee staff and partners. The interviews focus on coordination and decision-making; implementation challenges; contextual, systems, or other factors that affect implementation, and sustainability. If these data are collected less frequently, we will be less able to explain contextual factors affecting program implementation over time.
YFFG-Y YFFG-F	Information from the YFFG-Y and YFFG-F is designed to build an understanding of how youth and families in the AWARE and TISS programs describe their overall program experience and satisfaction with the programs. The focus groups will be conducted annually and include youth or parents representing a subset of AWARE and TISS grantees each year, such that all grantees will participate in the focus groups at least once during the evaluation. If these data are collected less frequently, we will be less able to understand youth and family perspectives about the programs across all grantees.
CPS	The CPS will collect information from AWARE and TISS grantees and representatives of their partner organizations about the characteristics of their partnerships and changes in policies related to the AWARE program. It is scheduled to collected once annually for three years. The consequence of collecting this data less frequently will be less ability to observe how partnerships and policies related to mental health change over the life of the project.
TSF	Grantees will complete the TSF on an ongoing basis to document the characteristics and participants of training provided as part of AWARE and TISS. The consequence of collecting this data less frequently will be lack of consistent information about trainings provided across grantees, limiting our information about the frequency and characteristics of the trainings provided through these programs.
PFF	A subset of participants in trainings provided through AWARE and TISS will be asked to complete the PFF after participating in trainings to gather information about their perception of the training and perceived impact. Data will be collected on an ongoing basis. The consequence of collecting this information less frequently is decreased information about the perceived quality of the trainings provided through AWARE and TISS and participants’ expectations about use of what was learned.
APPTS	A subset of participants in trainings provided through AWARE will be asked to complete the APPTS surveys before and after participating in trainings. Data will be collected on an ongoing basis. The consequence of collecting this information less frequently will be less ability to monitor changes in knowledge and capacity related to mental health care following trainings.
TPPTS	A subset of participants in trainings provided through TISS will be asked to complete the TPPTS surveys before and after participating in trainings. Data will be collected on an ongoing basis. The consequence of collecting this information less frequently will be less ability to monitor changes in knowledge and capacity related to trauma-informed care following trainings.
WFS	A subset of participants in trainings provided through AWARE and TISS will be asked to complete the WFS at 3 months and 12 months after participating in trainings to gather information about how information from the training was implemented, barriers to using what was learned, and any referrals to care made after the training. The consequence of collecting this information less frequently would be decreased information about how practice changed in the months after training participation and the extent to which those changes are maintained over time.
PCSS	The PCSS will be used to collect information from parents of students at schools participating in AWARE and TISS about school climate, safety, and supports for mental health. It is scheduled to be collected at two time points, once in year one and once in year three. The consequence of collecting this information less frequently would be to limit the information available about the school setting from parents’ perspectives.
STCSS	The STCSS will be used to collect information from students at schools participating in AWARE and TISS about school climate, safety, and supports for mental health. It is scheduled to be collected at two time points, once in year one and once in year three. The consequence of collecting this information less frequently would be to limit the information available about the school setting from students’ perspectives.
SSCSS	The SSCSS will be used to collect information from staff and teachers at schools participating in AWARE and TISS about school climate, safety, and supports for mental health. It is scheduled to be collected at two time points, once in year one and once in year three. The consequence of collecting this information less frequently would be to limit the information available about the school setting from staff members’ perspectives.
SIRF	The SIRF requires AWARE and TISS grantees to share existing data on students identified as in need of additional mental health or trauma-related support, including information on how the student was identified, whether and to which services they were referred, and the types of care received following referral. Data from the SIRF is integral to understanding the effectiveness of AWARE and TISS programs in improving the ability of schools to identify and intervene with students in need of behavioral health support, as well as increasing student access to care. Collecting this data on an ongoing basis is essential to capture changes in student behavioral health needs, identification pathways, and available referral resources throughout the course of AWARE or TISS funding.

2. Consistency with the Guidelines of 5 CFR 1320.5(d)(2)

The data collection fully complies with the requirements of 5 CFR 1320.5(d) (2).

2. Federal Register Notice and Consultation Outside the Agency

SAMHSA published a notice in the Federal Register January 10, 2024 (Volume 89, page 1581–1583), soliciting public comment on this study. No public comments were received.

Consultation on the design and instrumentation for the evaluation has occurred with individuals outside of SAMHSA. Most recently, in May 2023, we convened introductory webinars/feedback sessions with current Project AWARE and TISS grantees and EAP members that allowed us to discuss and finalize the current data collection plan and instruments. These consultations had four purposes: (1) to ensure continued coordination of related activities, especially at the Federal level; (2) to ensure the rigor of the evaluation design, the proper implementation of the design, and the technical soundness of study results; (3) to verify the relevance and accessibility of the data to be collected; and (4) to minimize respondent burden.

Consultation with the EAP will continue annually and as needed throughout the grant-funding period. Representatives on the EAP include leaders in the field of youth mental health program implementation, research, and evaluation. A Youth Advisory Board (YAB) will also be established that will be comprised of youth/and or families in the AWARE and TISS programs. The YABs will provide formative feedback related to youth engagement in the evaluation and culturally appropriate, resonant reporting.

2. Payment to Respondents

As with previous evaluations, the Project AWARE-TISS Cross-Site Evaluation will use a research-based approach and will require participation by youth and families, key partners, school staff, and grantee training participants. Renumeration is suggested for respondents at the time of their participation in surveys and interviews as compensation for the additional burden, potential inconvenience of participation, and any related costs (e.g., mobile phone minutes or data, compensation for time). Given the use of longitudinal data collection for the Project AWARE-TISS Cross-Site Evaluation and the hard-to-reach nature of these populations, compensation will be provided for data collection activities.

Planned remuneration for survey participants is a $20 incentive for participation in the process or outcome data collection activities including the IS, CPS, PCSS, STCSS, SSCSS, APPTS, TPPTS, and WFS; $10 will be provided to each respondent for completion of the PFF; and $25 will be provided for qualitative data collection using the IKII, YFFG-Y, and YFFG-F. Respondents to the SIRF and TSF are staff of the AWARE or TISS programs. Therefore, no remuneration is planned for these activities.

2. Assurances of Confidentiality

Data will be kept private to the extent allowed by law. To ensure the confidentiality of data collected and the protection of human subjects, the data collection protocol and instruments for the Project AWARE-TISS Cross-Site Evaluation will be reviewed through the ICF institutional review board (IRB) prior to the collection of covered or protected data. The ICF IRB holds a Federal wide Assurance (FWA00002349; Expiration, October 13, 2025) from the HHS Office for Human Research Protections (OHRP). This review ensures compliance with the spirit and letter of HHS regulations governing such projects. All protected data will be stored on secure servers in the SAMHSA. In addition, the web-based data collection and management system, the ATODS, will facilitate data entry and management for the evaluation.

Descriptive information will be collected from respondents during data collection activities. Quantitative data collection will occur via the ATODS portal, however if some activities require hard copy forms (or grantees request hard copy forms) with identifying information, these will be stored in locked cabinets. The contact information will be entered into the ATODS via a page on the site that can only be accessed by the limited number of individuals who require access (selected ICF staff such as data analysts and administrative staff). These individuals have signed privacy, data access, and data use agreements. Identifying information collected to facilitate the administration of surveys will not be stored with survey responses. Further, datasets will be stripped of any identifying information prior to use by data analysts. Once data collection is concluded and incentives are distributed, respondent contact information will be deleted from the database and the hard copy forms will be destroyed.

Data collection activities requiring the collection of identifying information for the AWARE-TISS Cross-Site Evaluation include the following: APPTS, TPPTS, IKII, YFFG-Y, and YFFG-F. Specific procedures to protect the privacy of respondents are described below.

• APPTS and TPPTS: As part of each AWARE or TISS training event, AWARE and TISS training participants will be asked to complete a pre-post training survey. At the end of these surveys, there is a consent-to-contact request. The consent-to-contact request is included as part of the APPTS or TPPTS that is completed via a QR code or URL. The consent-to-contact request will ask trainees to provide identifying information (i.e., name, email, phone number) necessary to contact them and will ask for their consent to be contacted again to answer some brief, additional follow-up questions using the PFF and WFS. The PFF will gather additional information about training and the WFS will gather information about longer-term impact on systems and communities as well as barriers and supports to training 3- and 12-months post training.

• IKII: Identifying information for respondents to the IKIIs will be necessary for scheduling purposes. Although grantees will coordinate recruitment, ICF will coordinate scheduling and conduct interviews. Contact information will be limited to key informant names, email addresses, and phone numbers and will be entered into a password-protected database. These will be kept separate from de-identified interview transcripts and only used for interview scheduling purposes. No identifying information will be used during the interview.

• YFFG-Y and YFFG-F: Identifying information for respondents to the YFFG-Y and YFFG-F will be necessary for scheduling focus groups. Although grantees will coordinate recruitment, ICF will coordinate scheduling and facilitate focus groups. Contact information will be limited to youth and parent (or family/caregiver) names, email addresses, and phone numbers and will be entered into a password-protected database. These will be kept separate from de-identified focus group transcripts and only used for focus group scheduling purposes. No identifying information will be used during the focus group.

2. Questions of a Sensitive Nature

Survey and interview instruments include questions that are potentially sensitive because this project concerns the topic of mental health and connection to positive health supports. These questions collect information about mental health in general, mental health challenges (e.g., bullying, substance use, mental illness, suicide), family circumstances, and mental health services and programs. These questions are central to the agency’s goal of learning about the identification of at-risk youth, understanding the referrals and services they receive and understanding the youth experience. Names and email addresses collected as part of the scheduling process (for interviews and focus groups) will be kept separate from responses as stated above. All data will be managed and stored in the manner described above and therefore will be available only to authorized evaluation staff. Active consent/assent forms explicitly advise potential respondents and participants if there is potentially sensitive content as well as the voluntary nature of all data collection activities. Unanticipated or negative consequences will be reported immediately to grantee/local and ICF institutional review boards (IRB). If any negative consequences do result, the Principal Investigator and Project Director will consult with appropriate clinical professionals and determine the suitable action and make the appropriate referrals.

2. Estimates of Annualized Burden Hours and Costs

Clearance is being requested for three years of data collection for the Project AWARE-TISS Cross-Site Evaluation. Exhibit 4 displays data collection instruments, estimated completion time (in minutes), respondent number/type, and frequency of completion.

Exhibit 4. Overview of Data Collection Instruments by Completion Time, Respondent Number/Type, and Frequency of Completion

Instrument	Estimated Completion Time	Number and Type of Respondents	Frequency of Completion
IS	30 minutes	128 AWARE project coordinators (4 per AWARE grantee) 15 TISS project coordinators (1 per TISS grantee) 15 TISS program staff (1 per TISS grantee)	Once annually in years 1-3
IKII	60 minutes	47 state project coordinators (1 per grantee) 47 local project coordinators (1 per grantee) 47 community mental health providers (1 per grantee) 94 school mental health providers (2 per grantee) 47 school administrator (1 per grantee)	Once annually in years 1-3
YFFG-Y YFFG-F	90 minutesa	Focus groups to include up to 10 youth or family participants in each; focus groups for 12 AWARE grantees per year and 6 TISS grantees per year such that all grantees have focus groups at least once during the 3-year data collection period.	Once annually in years 1-3
CPS	15 minutes	128 AWARE project coordinators (4 per AWARE grantee) 32 community mental health providers (1 per AWARE grantee) 32 AWARE school administrators (1 per AWARE grantee) 15 TISS project coordinators (1 per TISS grantee) 15 TISS mental health providers (1 per TISS grantee) 15 TISS school administrators (1 per TISS grantee)	Once annually in years 1-3
TSF	9 minutes	47 grantee program staff (10 per grantee)	Ten timesb annually in years 1-3 year
PFF	9 minutes	2,400 AWARE training participants (75 per AWARE grantee) 375 TISS training participants (25 per TISS grantee)	Once annually in years 1-3
APPTS TPPTS	30 minutesc	4,000 AWARE training participants (125 per AWARE grantee) 750 TISS training participants (50 per TISS grantee)	Once annually in years 1-3
WFS	15 minutes	2,016 participants from (63 per AWARE grantee) 375 training participants (25 per TISS grantee)	Twice annually in years 1-3
PCSS STCSS SSCSS	24-30 minutes	282 students (6 per grantee) 282 parents (6 per grantee) 282 school staff (6 per grantee) 188 school district administrators (4 per grantee)	Once annually in year 1 and year 3
SIRF	30 minutes	47 grantee program staff (100 per grantee)	Up to 100d times annually in years 1-3

^a Youth and family focus groups are estimated to last between 60-90 minutes; this takes into consideration any logistical needs prior to the discussion starting (e.g., bathroom break, consent/assent, setting expectations) plus the time for the discussion.

^b It is estimated that the TSF will be completed by grantee program staff following each training; an average of 10 trainings annually was used for the estimate as grantees are required to develop a training plan and conduct trainings for teachers, teacher assistants, school leaders, specialized instructional support personnel, and mental health professionals.

^c Estimated completion time considers total completion time for the pre- and post-survey (15 min for pre, 15 min for post)

^d The SIRF will be completed for up to 100 youth per grantee per year. This number could be much less depending on the number of referrals that are made.

Exhibit 5 describes the estimated annualized burden and costs associated with data collection activities. Burden is calculated for data collection that will span across 47 AWARE/TISS grantees (32 AWARE, 15 TISS), which represents the number active grantees anticipated for each year of data collection. The cost was calculated based on the hourly wage rates for appropriate wage rate categories using data collected as part of the Occupational Employment Statistics Survey (BLS, 2022) and from the U.S. Department of Labor Federal Minimum Wage Standards. Exhibit 6 shows an annualized summary of burden hours by respondent type. Annual Burden (hours) and Total Cost ($) are rounded to the nearest whole number.

Exhibit 5. Estimated Annualized Burden Hours and Costs (Across the 3-Year Clearance Period)

Instrument	Type of Respondent	Number of Respondents	Responses per Respondent	Total Number of Responses	Burden per Response (hours)	Annual Burden (hours)	Hourly Wage Rate ($)	Total Cost ($)
IS	Project Coordinator	143	1	143	0.5	72	$35.522	$2,557
IS	Program Staff	15	1	15	0.5	8	$21.713	$174
IKII	Project Coordinator	94	1	94	1	94	$35.52	$3,339
IKII	Mental Health Provider	141	1	141	1	141	$69.394	$9,784
IKII	School Administrator	47	1	47	1	47	$54.215	$2,548
YFFG-Y	Youth	79	1	79	1.5	119	$7.256	$863
YFFG-F	Parent of Youth	79	1	79	1.5	119	$7.25	$863
CPS	Project Coordinator	143	1	143	0.25	36	$35.52	$1,279
CPS	Program Staff	47	1	47	0.25	12	$21.71	$261
CPS	School Administrator	47	1	47	0.25	12	$54.21	$651
TSF	Program Staff	47	10	470	0.15	71	$21.71	$1,541
PFF	Program Trainee	2,775	1	2,775	0.15	416	$26.817	$11,153
APPTS	Program Trainee	4,000	2	8,000	0.25	2,000	$26.81	$53,620
TPPTS	Program Trainee	750	2	1,500	0.25	375	$26.81	$10,054
WFS	Program Trainee	2,391	2	4,782	0.25	1,196	$26.81	$32,065
PCSS	Parent of Youth	282	1	282	0.4	113	$7.25	$819
STCSS	Youth	282	1	282	0.4	113	$7.25	$819
SSCSS	School Staff	282	1	282	0.5	141	$30.208	$4,258
SSCSS	School Administrator	188	1	188	0.5	94	$54.21	$5,096
SIRF	Program Staff	47	100	4,700	0.5	2,350	$21.71	$51,019
	Total	11,879		24,096		7,529		$192,763

Exhibit 6. Annualized Summary Burden by Respondent Type (Across the 3-Year Clearance Period)

Respondents	Number of Respondents	Responses/ Respondent	Total Responses	Total Annualized Hour Burden
Project Coordinator	427	3	380	202
Program Staff	109	112	5,232	2,441
Program Trainee	9,916	7	17,057	3,987
Mental Health Provider	141	1	141	141
School Administrator	282	3	282	153
School Staff	282	1	282	141
Parent of Youth	361	2	361	232
Youth	361	2	361	232
Total	11,879	131	24,096	7,529

2. Estimates of Annualized Cost Burden to Respondents or Record Keepers

Grantees are collecting the majority of the required data elements as part of their normal Project AWARE and TISS program operations. Grantees maintain this information for their own program planning, quality improvement, and reporting purposes. Therefore, there are no additional capital or start-up costs associated with the AWARE-TISS Cross-Site Evaluation. There will be some additional burden on record keepers to provide potential respondent lists for data collection activities. However, these operation costs will be minimal. Each grantee has been funded, as part of the overall cooperative agreement award, to participate in the requirements of the Project AWARE-TISS Cross-Site Evaluation. Therefore, no cost burden is imposed on the grantee by this additional effort.

2. Estimates of Annualized Cost to the Government

CMHS has planned and allocated resources for the management, processing and use of the collected information in a manner that will enhance its utility to grantees, the government, community agencies, and the public. Including the Federal contribution to local grantee evaluation efforts, the contract with the ICF (the national evaluator), and Government staff to oversee the evaluation, the annualized cost to the Government is estimated at $2,511.28. These costs are described below.

A contract has been awarded to ICF for evaluation of the Project AWARE-TISS Cross-Site Evaluation. The current evaluation contract with SAMHSA is funded to conduct the evaluation with 47 grantees over the next 4 years with a value of $9,943,116. The estimated average annual cost of the contract will be $2,485,779. This covers expenses related to developing and monitoring the AWARE-TISS Cross-Site Evaluation including, but not limited to developing the evaluation design and instrumentation; developing training and technical assistance resources (e.g., manuals, training materials); conducting training and technical assistance; monitoring of grantees; traveling to grantee sites and relevant meetings; and analyzing data and disseminating findings. In addition, these funds will support the maintenance of the web-based data collection and management system and fund staff support for data collection. It is estimated that CMHS will allocate 0.30 of a full-time equivalent each year for Government oversight of the evaluation. Assuming an annual salary of $85,000, these Government costs will be $25,500 per year.

2. Changes in Burden

SAMHSA is requesting 7,529 annual burden hours for this submission over the three-year clearance period.

2. Time Schedule, Publication, and Analysis Plans

1. Time Schedule

The time schedule for implementing the Project AWARE-TISS Cross-Site Evaluation is summarized in Exhibit 7. A 3-year clearance is requested for this project.

Exhibit 7. Time Schedule

Primary Data Collection Activities by Evaluation Study	Timeframe
	2024	2025	2026
Implementation and Sustainability Study
IS
IKII
YFFG-Y, YFFG-F
Systems Change Study
CPS
PCSS, STCSS, SSCSS
TSF
PFF
APPTS, TPPTS
WFS
Identification and Referral Study
SIRF
Youth Resiliency and Outcomes Study
YFFG-Y

2. Publication Plans

The Project AWARE-TISS Cross-Site Evaluation contract with ICF requires ongoing quarterly, annual, and ad hoc reports summarizing the results and progress of the evaluation. The evaluation team will analyze collected data and prepare reports to summarize key findings. A final report on the results of the Project AWARE-TISS Cross-Site Evaluation is also required and will be produced by the evaluation team. Due to the importance of this work to the field of behavioral health, in collaboration with SAMHSA and the Government project officer, the results of the evaluation may also be published in relevant professional journals to inform the research community as well as the decision making of policymakers and program administrators.

Potential manuscript topics will include findings related to priority areas such as the impact of the AWARE and TISS programs on youth morbidity, the care pathways that youth follow after at-risk identification, and the ability of grant program trainees to retain the needed skills to be effective in their roles. Additional manuscript topics may also be related to the research questions and findings that emerge from individual process evaluation, outcome evaluation, or cross-program studies. All publications will be submitted to the Contracting Office Representative (COR) in draft form for review and approval prior to submission to the selected journal. Examples of journals that may be considered as vehicles for publication include the following:

• American Journal of Public Health

• American Journal of School Health

• Child Development

• Journal of the American Academy of Child and Adolescent Psychology

• Journal of Child and Family Studies

• Journal of Clinical Child and Adolescent Psychology

• Journal of Health and Social Behavior

• Journal of Mental Health Administration

• Psychological Reports

3. Data Analysis Plan

Data collected through the Project AWARE-TISS Cross-Site Evaluation will be analyzed to address key evaluation questions and related sub questions. Analysis plans for each evaluation study, program-specific sub-study and cross-program analyses are described below.

1. Process Evaluation Analysis

Implementation and Sustainability Study

Through primary and secondary data collection and drawing on the range of data sources noted, including document and extant data extraction, the IS survey and interviews, and focus groups with youth/parents/families, we will provide a comprehensive assessment of current AWARE and TISS grantee implementation activity, progress, and context.

To assess the extent to which grantees are implementing the various aspects of the AWARE and TISS programs, we will analyze secondary data from grantee documents and reports using descriptive statistics to provide a precise characterization of the related strategies and activities and their outputs. This will include, for example: (1) types of activities implemented including evidence-based practices and innovative approaches, (2) populations of focus, (3) implementation challenges and facilitators, (4) outputs such as number of participants served, (5) grantee sustainability plans, and (6) whether grantees are implementing strategies as intended and in accordance with their work plan. Building on the findings from grantee document review, the IS survey assesses program implementation and sustainability from a state, local, and community perspective, helping to contextualize common and varying experiences across implementation settings, grantee sites, demographics, school ecologies, partner types, and programs. ICF will merge data from primary and secondary sources across multiple levels (e.g., state, county) into comprehensive datasets for analysis and use descriptive statistics to summarize this information across grantees. When multiple data sources address the same evaluation questions, we will perform validation checks to ensure reliable and valid reporting.

Further, to obtain rich, in-depth information and description related to implementation experiences, and prioritize the voices of program partners and consumers, we have included KIIs to supplement the survey findings and focus groups with youth to explore their experience with the AWARE and TISS programs and related services. We will analyze qualitative data generated from the document review, interviews, and focus groups for themes, patterns, and interrelationships relevant to the evaluation questions. Textual data from interview and focus group transcripts will be entered into a qualitative database software program, MAXQDA, for analysis. The team will work with the SAMHSA COR and other stakeholders to develop an initial list of deductive codes aligned with the study questions and systematically code the data to identify relevant themes in preparation for unique and common thematic analyses. The intention is to specifically explore, among other factors, the implementation efforts and innovations resulting from the AWARE and TISS grant programs overall and to meet the needs of subpopulations and under-resourced communities. As the team begins to generate conclusions about the data during coding, we will verify these more general analyses and validate them by cross-checking and revisiting the data. Thus, the coding scheme will be elaborated upon and refined based on themes that emerge from the data. The descriptive themes that emerge from the synthesis of findings will be discussed at length with the coders and the larger evaluation team to ensure the validity of the conclusions.

Systems Change Study

Descriptive statistics will be used to summarize the data collected using the CPS, PCSS, STCSS, SSCSS, TSF, PFF, APPTS, TPPTS, and WFS to address questions related to the number and characteristics of partnerships at each time point, including numbers of partners from different sectors. Cronbach’s alpha will be calculated to assess the internal consistency of scales being used. To assess change over time on the training pre- and post-surveys, multivariate repeated measures analysis of variances will be used.

2. Outcome Evaluation Study Analysis

Identification and Referral Study

To understand the proportion of youth who received referrals and services, ICF will use descriptive statistics to regularly provide a precise characterization of AWARE and TISS identification and referral activities, as well as youth engagement in care following referral. This includes 1) the number, characteristics (e.g., gender, race/ethnicity) of youth identified, and the settings where they were identified; 2) the proportion of youth who received a referral following identification; and 3) the proportion of youth who received follow-up support and ongoing treatment up to 3 months after initial identification. We will also use descriptive techniques to understand which individuals within a school-based setting are most likely to identify youth, where and to which services youth are most often referred, and service engagement rates. We will assess the extent to which grantees are able to identify and link school-aged children and youth to the appropriate care by analyzing variation over time and across grantee sites within each program using mixed-effects regression models. These models will explore the association between outcomes (the number of youths screened, identified as in need of services, and receiving mental health services/supports) and characteristics of youth identified, grantee program implementation, and the structure and capacity of the referral system (i.e., MOUs and policies established, school staff training outcomes). Data from the Identification and Referral Study will also be used in the quasi-experimental comparison study described in the Cross-Program Impact Analysis section.

Youth Resiliency and Outcomes Study

Data from disparate data sets will be merged into an analytic data set. Quantitative data will be analyzed using statistical software such as R or Stata. The team will use several different approaches including descriptive and inferential analyses to assess changes in school and student outcomes after participation in the AWARE or TISS programs. ICF will use descriptive analyses to understand contextual and demographic information across grantees and use repeated measures analyses to examine changes over time. Repeated measures analyses will be conducted with NOMs data, SEA/LEA administrative data, and state or local-level survey data. Variation in performance measures over time will be analyzed using mixed-effects regression models or generalized estimating equations.

Qualitative data collected from grantee reports and student focus groups will provide meaningful context about school and student outcomes and will be analyzed using thematic coding. ICF will develop coding rubrics and codebooks through an iterative coding process. Our team will develop composite scoring, categorization, and comparison techniques using a traditional inductive coding method with well-established procedures for coding and analyzing these data. This codebook will be uploaded to a qualitative data analysis program (e.g., MAXQDA) and used to code all focus group transcripts and establish interrater reliability. ICF will identify common themes that emerge across participant descriptions.

3. Program-Specific Sub-Studies

TISS Case Studies

ICF proposes a concurrent triangulation analysis approach in which qualitative and quantitative data are collected and analyzed separately, and then combined at later stages to address different dimensions of the same overarching evaluation question (Castro et al., 2010). This approach generates more thorough and complete answers to complex social phenomena that operate across multiple units of analysis and time periods. The team will analyze qualitative data gathered using similar analytic techniques and codebook development as employed in the Implementation and Sustainability Study. Using qualitative software (e.g., MAXQDA) we will analyze textual data to understand similarities and variations in grantee implementation processes, challenges and successes experienced, and other contextual factors. Our team will code qualitative interview data by developing coding rubrics and codebooks. We will develop composite scoring, categorization, and comparison techniques using a traditional inductive coding method with well-established procedures for coding and analyzing the data. The thematic analyses will also provide a context in which quantitative data from the outcomes study can be interpreted. For example, challenges related to recruitment and retention in services may be better understood with insight related to the barriers to implementation emerging in themes from the qualitative analysis.

AWARE Suicide Awareness and Prevention Sub-Study

Descriptive statistics will be used to provide a precise characterization of suicide awareness and prevention training activities, their outputs, and proximal outcomes. This includes (1) the number of suicide awareness and prevention training policies implemented; (2) the structure and characteristics of suicide awareness and prevention training policies implemented; (3) the number of suicide prevention and awareness trainings conducted; and (4) students help-seeking reports.

Cross-Program Impact Analyses

Cross-Program Impact Analysis

The Cross-Program Impact Analysis will provide a quasi-experimental assessment of whether key outcomes in the AWARE/TISS sites are different from what would have been observed in the absence of the programs. The counties with the AWARE/TISS grantees will be compared to non-intervention counties to examine the impact of program activities on key outcomes using synthetic control methodology (SCM). SCM uses a weighted combination of potential control cases to represent what would have happened in treated cases without an intervention. Potential control cases must be like the treated cases and must not have been exposed to an intervention. For example, the CDC Detailed Mortality Files can be used to provide geocoded suicide mortality rates for the grantee locations where the programs were implemented. The remaining cases serve as the potential pool of control cases. The predictor variables used to match the real and synthetic treated cases are the factors theoretically associated with the outcome variable. The synthetic control (SC) is built from the values of the outcome and predictor variables of the potential controls usually made up of a geographical grouping of cases (e.g., counties) prior to the implementation of the intervention in the grantee sites. The analysis then calculates a weighted average of the control cases. The aim is to minimize the difference in the outcome trends between the intervention cases and the SC cases during the pre-intervention period. Once an acceptable SC has been obtained between actual and synthetic cases, the outcome analysis can be run using the full pre- and post-implementation data.

Secondary data will be used for the quasi-experimental comparison study of select AWARE and TISS grantee sites. We will use SCM for sites with available extant data constructing artificial sites based on data from several counties not participating in the grant programs. These artificial sites will resemble the AWARE and/or TISS sites in both pre-intervention values of the outcomes and predictors of the outcome variable. For each intervention site and outcome of interest, the SC is a weighted average of sites from the control pool, with weights chosen to match the treated state's pre-treatment outcomes as closely as possible (Abadie et al., 2010, 2015; Abadie & Gardeazabal, 2003). The estimated impact is then the difference in post-treatment outcomes between the treated unit and the SC. We will estimate the impact of the AWARE/TISS programs using SCM. We will use machine learning to identify different patterns of effect and contextual characteristics that may predict them using Bayesian additive regression tree (BART). An ensemble of tree models will be used to flexibly model the observed outcome. Bayesian priors will be used to minimize identification of large trees and to give small weight to any singular tree in the ensemble. Machine learning procedures will be used to identify combinations of characteristics and circumstances from a potentially large set of alternatives, as well as their potential interactions, without relying on stringent functional form assumptions of traditional methods like stepwise regression. We will use recursive partitioning algorithms to split the data repeatedly to identify subgroups that are homogeneous with respect to the impact variable. Specifically, the so-called “evolutionary algorithm” has been demonstrated to have comparatively better predictive accuracy than methods based on forward stepwise search (Grubinger et al., 2014). Further, by applying the same analysis techniques to related variables that are not expected to be affected by AWARE/TISS interventions (i.e., ‘control’ outcomes), we can gain added protection against unmeasured confounding variables.

Behavioral Health Equity Cross-Study Analysis

The Behavioral Health Equity Cross-Study analysis will apply innovative models to assess the impact of the program on subpopulations with disparate mental health outcomes. Innovative data science methods exist that can be used to inform the mechanisms of health disparities to guide intervention development. Existing publicly available data related to mental health incidence, policies, population characteristics, and other domains available over time can be combined to provide information at fine spatial and temporal scales to inform targeted interventions (Breen et al., 2019). Evaluation of these data could identify areas with disparities, changes in disparities over time, and factors associated with disparities (Zhang et al., 2017).

To address the question of potential behavioral health disparities in the impact of the AWARE and TISS programs on the specific subgroups, we propose spatiotemporal models for impact evaluation. Assessing the impact of an intervention (such as a project, program, or policy) in small geographic areas or specific subgroups with small population sizes can be challenging where reliable estimates of the outcome of interest are difficult to obtain. An area or domain of estimation is small precisely when direct estimations are extremely unreliable or entirely unfeasible give the sample size. The situation often arises with relatively rare outcomes (such as suicide), as well as with more frequent outcomes (such as substance use), when the interest lies in specific segments of the population. The field of disease mapping is concerned with estimating the risk of a disease or health outcome using case counts within small administrative districts or regions. Building on hierarchical models originally proposed for small-area estimation, we have used disease-mapping modeling techniques that incorporate spatial dependence and take advantage of multiple time periods to address this challenge. The introduction of Bayesian spatiotemporal models developed for disease mapping (Bauer et al., 2016) takes advantage of spatial structure together with temporal dependencies to aid estimation. The application extends the utility of the approach beyond disease mapping and significantly advances the ability to understand the impact of grant-funded programs in high-risk populations. We will use spatiotemporal models for impact evaluation combining the use of small-area estimation, conditional autoregressive modeling, and integrated nested Laplace approximation with the quasi-experimental approach described in the Cross-Program Impact Analysis section that involves using SCM and refining the solution with machine learning techniques.

17. Display of Expiration Date

All data collection instruments will display the expiration date of OMB approval.

18. Exceptions to the Certification Statement

This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.

1 School climate scales were not created for the parent survey.

2 BLS OES May 2022 National Industry-Specific Occupation Employment and Wage Estimates average annual salary for Community and Social Service Specialists, All Other (code 21-1099); https://www.bls.gov/oes/current/oes_nat.htm#21-0000

3 BLS OES May 2022 National Industry-Specific Occupation Employment and Wage Estimates average annual salary for Community and Social Service Assistants (code 21-1093); https://www.bls.gov/oes/current/oes_nat.htm#21-0000

4 BLS OES May 2022 National Industry-Specific Occupation Employment and Wage Estimates average annual salary for Healthcare Diagnosing or Treating Practitioners (code 29-1000); https://www.bls.gov/oes/current/naics5_541720.htm#29-0000

5 BLS OES May 2022 National Industry-Specific Occupation Employment and Wage Estimates average annual salary for Educational Administrators, All Other (code 11-9039); https://www.bls.gov/oes/current/naics5_541720.htm#11-0000.

6 https://www.usa.gov/minimum-wage

7 BLS OES May 2022 National Industry-Specific Occupation Employment and Wage Estimates average annual salary for Community and Social Service Occupations (code 21-0000); https://www.bls.gov/oes/current/oes_nat.htm#21-0000

8 BLS OES May 2022 National Industry-Specific Occupation Employment and Wage Estimates average annual salary for Educational, Guidance, and Career Counselors and Advisors (code 21-2012); https://www.bls.gov/oes/current/naics5_541720.htm#21-0000

12