SS Part A 12.15.15

National Outcomes Evaluation

Garrett Lee Smith (GLS) Suicide Prevention Program

Supporting Statement

1. Justification

The Substance Abuse and Mental Health Services Administration’s (SAMHSA’s) Division of Prevention, Traumatic Stress and Special Programs of the Center for Mental Health Services (CMHS) is requesting clearance for the revision of data collection associated with the previously-approved cross-site evaluation of the Garrett Lee Smith (GLS) Youth Suicide Prevention and Early Intervention Program (GLS Suicide Prevention Program), now entitled National Outcomes Evaluation (NOE). Passed by Congress in 2004, the Garrett Lee Smith Memorial Act (GLSMA) was the first legislation to provide funding for States, Tribes, and institutions of higher education to develop, improve, and evaluate early intervention and suicide prevention programs. In addition to providing programmatic funding, the GLSMA mandates that the effectiveness of the GLS Suicide Prevention Program be evaluated and reported to Congress. Per this mandate, the cross-site evaluation of the GLS Suicide Prevention Program was conceptualized in 2005 and has been implemented since. As a result of the vast body of information collected and analyzed through the cross-site evaluation, SAMHSA has identified areas for additional investigation and the types of inquiry needed to move the cross-site evaluation into its next phase.

Informed by its nine-year history partnering with State/Tribal and Campus grantees, evolution of the GLS Suicide Prevention Program, and findings from the nine-year cross-site evaluation, SAMHSA is embarking on the next phase of the evaluation. The National Outcomes Evaluation—a proposed redesign of the currently-approved cross-site evaluation (OMB No. 0930-0286; Expiration, January 31, 2017)—builds on prior published GLS evaluation proximal and distal training and aggregate findings from program activities (e.g., Condron et al., 2014; Walrath et al., 2015). The updated design reflects SAMHSA’s desire to assess the implementation, outcomes, and impacts of the GLS program across its two components—the GLS State/Tribal Program and the GLS Campus Program. As such, the NOE is designed to address the field’s need for additional evidence on the impacts of the GLS Suicide Prevention Program in three areas:

• Suicide prevention training effectiveness

• Early identification and referral on subsequent care follow-up and adherence

• Suicide safer care practices within health care settings

Clearance is being requested for data collection associated with 11 instruments—specifically, 6 revised, 2 new, and 3 existing instruments. These include Web-based surveys, inventories, and forms; telephone surveys (using CATI technology); mobile telephone text-message surveys; and abstractions/submissions of existing data. As before, some instruments will apply to both or either State/Tribal and/or Campus programs only. Due to the fulfillment of data collection requirements, approval for removal of 6 instruments is also requested. In addition, data collection requirements for 3 instruments will be fulfilled during year 1 of the OMB collection period, after which the instruments will be discontinued. (See Section A.2.a for a description of data collection activities and Section A.2.b for revisions to the evaluation.)

1. Circumstances of Information Collection

   1. Background

Suicide continues to be a major public health problem in the United States. In 2010 (the latest data available), suicide was the 10th leading cause of mortality, claiming more than 38,000 lives, including 4,867 youth aged 10–24 (CDC National Center for Injury Prevention and Control, [CDCNCIPC], 2010), and resulting in an economic cost estimated to be $34 billion (American Foundation for Suicide Prevention, 2010). Moreover, for every suicide death, there are approximately 25 attempted suicides (Crosby et al., 2011). Rates of suicide and suicidal ideation are even higher among certain subgroups, especially young Native Americans and Alaska Natives (CDC, 2012c; Goldston et al., 2008); Hispanic females (Substance Abuse and Mental Health Services Administration [SAMHSA], 2005); and lesbian, gay, and bisexual youth (Suicide Prevention Resource Center [SPRC], 2008). Youth who exhibit risk factors, such as depression, impulsivity, alcohol and substance abuse, and a history of trauma or abuse, are believed to have a greater potential for suicidal behavior (Cash, 2009). Suicide is also a leading cause of death for college students (King, Vidourek, & Strader, 2008), with 11% of screened college students suffering from recent or current suicidal ideation (Garlow et al., 2008). In a study by the American College Health Association (Reference Group Executive Summary, Spring 2012), 47% of college students reported feeling hopeless, 32% reported feeling so depressed they could barely function, and 8% reported feeling suicidal. In addition, suicide rates are higher among some veteran populations compared with the general population (U.S. Department of Health and Human Services [HHS], 2012).

Over the past 15 years, national leaders and experts have responded to the public health crisis of suicide by bringing attention and committing resources to this preventable tragedy. Three documents, Reducing Suicide: A National Imperative (Institute of Medicine, 2002), The Surgeon General’s Call to Action to Prevent Suicide (HHS, Public Health Service [PHS], 1999), and the National Strategy for Suicide Prevention (National Strategy): Goals and Objectives for Action (HHS, PHS, 2012), all provide overlapping recommendations for how this problem can be effectively addressed. The President’s New Freedom Commission on Mental Health (2003) was followed by passage of the landmark GLSMA (P.L. 108-355)—the largest Federal effort ever focused specifically on addressing the tragedy of youth suicide. The GLSMA authorized the use of $82 million over three years to support the GLS State/Tribal and the GLS Campus Programs. These grant portfolios support suicide prevention programming for youth aged 10–24 throughout the United States, U.S. territories, tribal communities, and campuses. The GLSMA also authorized the establishment of the Suicide Prevention Resource Center (SPRC), which provides programmatic training and technical assistance (TTA) to grantees and suicide prevention stakeholders.

1. GLS State/Tribal Program

The focus of the GLS State/Tribal Program is to support the development and implementation of statewide or tribal youth suicide prevention and early intervention strategies, with emphasis on public/private collaborations with youth-serving institutions and agencies. State/Tribal Program activities include outreach and awareness initiatives, school and community-based gatekeeper trainings, clinical trainings for mental health professionals and hotline staff, screening programs, means restriction campaigns, policies and protocols related to intervention and postvention, coalitions and partnerships, direct mental health, postvention, case management, crisis response services, and traditional healing practices. Although previously awarded for 3 years of funding, beginning in 2014, grants are awarded for 5-year periods. State/Tribal grantees serve priority populations including lesbian, gay, bisexual, transgender, and questioning (LGBTQ) youth, American Indian and Alaska Native (AI/AN) residents, youth in contact with juvenile justice systems, military family members, veterans, and survivors of suicide attempts and those who have experienced suicide loss. Since 2005, the SAMHSA GLS Suicide Prevention Program has awarded funding to 181 State, tribal, and territory grantees over nine cohorts. Within comprehensive and community-based systems, GLS State/Tribal grantees are expected to prioritize the following evidence-based practices and activities, with the goal of reducing rates of suicidal ideation, suicide attempts, and suicide deaths in their communities:

• Train persons in youth-serving organizations (e.g., schools, foster care systems, substance abuse programs, and juvenile justice programs) to identify and refer at-risk youth

• Train clinical service providers to assess, manage, and treat youth at risk for suicide

• Improve continuity of care and follow-up of youth identified at risk for suicide after discharge from emergency departments or inpatient psychiatric units

• Conduct outreach and awareness activities and promote the utilization of the National Suicide Prevention Lifeline (NSPL)

• Identify sources of timely surveillance data

• Form or participate in public/private coalitions and partnerships with youth-serving agencies

• Develop policies and protocols for identification, referral, and follow-up of youth at risk of suicide

2. GLS Campus Program

The GLS Campus Program focuses on preventing suicide in higher education by developing campus infrastructure and building capacity to support effective and sustainable mental health services for students. The grants are awarded for 3 years. Campus grantees must also prioritize the needs of high-risk populations, including LGBTQ students, AI/AN residents, military family members, and veterans. Program activities typically include public awareness and social marketing campaigns, outreach and awareness events and products, gatekeeper trainings, assessment and referral trainings for clinical staff, life skills and wellness activities, screening programs, coalitions, and crisis protocol development. SAMHSA has awarded 190 GLS Campus Program grants to 175 institutions of higher education over 8 cohorts. In partnership with other campus health and wellness initiatives, academic departments, administrations, and GLS State/Tribal grantees, GLS Campus grantees are expected engage in the following efforts:

• Train students, faculty, and staff on suicide prevention and mental health promotion

• Collaborate with campus and community partners to deliver outreach and awareness strategies

• Conduct educational seminars and distribute informational materials to students, faculty, staff, and family members on suicide prevention, identification, and reduction of risk factors (e.g., depression and substance use/abuse)

• Increase help-seeking behaviors and reduce negative attitudes surrounding mental health and substance abuse treatment among students

• Create local college-based hotlines or promote the NSPL

• Develop crisis response protocols

While the GLSMA paves the way for the development and enhancement of suicide prevention programming across the United States, it does not end there. Rather, the GLSMA further directs these programs to evaluate the effectiveness of their targeted interventions at the local level, and requires an evaluation and report to Congress. The cross-site evaluation, now the National Outcomes Evaluation, is designed to evaluate the effectiveness of suicide prevention activities and has been essential in helping communities and decision-makers at all levels of government to improve suicide prevention effectiveness.

2. The Need for Evaluation

Section 520E (g) of the GLSMA mandates a cross-site evaluation to be conducted concerning the effectiveness of the activities carried out under the GLS State/Tribal Program. The GLSMA specifies that a report to Congress must be submitted:

…to analyze the effectiveness and efficacy of the activities conducted with grants, collaborations and consultations under [Section 520E].

In addition, Section 520-E-2 (f) of the GLSMA mandates a cross-site evaluation of the GLS Campus Program. The GLSMA specifies that a report must be submitted to Congress to include:

…an evaluation of the grant program outcomes, including a summary of activities carried out with the grant and the results achieved through those activities. [including] recommendations on how to improve access to mental and behavioral health services at institutions of higher education, including efforts to reduce the incidence of suicide and substance abuse.

The purpose of the NOE is to expand upon information collected through the cross-site evaluation and document the process, outcomes, and impacts of the GLS Suicide Prevention Program. The NOE will serve as a primary mechanism through which the GLS Suicide Prevention Program will continue to be understood, improved, and sustained. A government contractor (ICF) will coordinate data collection for the evaluation and provide support for its local-level implementation. Each grantee is required by the cooperative agreement and grant to conduct a self-evaluation and to participate in the NOE. In this partnership, ICF provides training and technical assistance (TA) regarding data collection and research design for the evaluation. In addition, ICF directly collects data, receives data from grantee data collection efforts, monitors data quality, and provides feedback to grantees. Data gathered through the NOE will continue to be utilized for both grantee-specific and national assessments of the program.

The updated design for the NOE comprises three distinct studies—Training, Continuity of Care, and Suicide Safer Environment—that provide continuity and utility of data collected during the 9-year cross-site evaluation on the implementation and proximal outcomes of the GLS program. Two “enhanced” components will provide for key findings derived through experimental and quasi-experimental methods without undue burden on grantees and youth. The multimethod approach addresses the evaluation questions of the GLS Suicide Prevention Program, furthers the understanding of the impacts of the GLS Suicide Prevention Program, and meets the GLSMA legislative mandate for evaluating program performance.

3. Previously Approved Clearance

Currently, data collection for the cross-site evaluation is operating under OMB clearance (No. 0930-0286), valid through January 31, 2017. This section contains a brief description of the cross-site evaluation design outlined in the previously-approved OMB request. The cross-site evaluation collects and analyzes comprehensive data that focus on the context within which GLS programs are implemented, the products and services that are developed and utilized, the process through which programmatic activities are implemented, and impacts associated with those activities. To achieve this purpose, the cross-site evaluation consists of four stages of information gathering that target funded program activity areas: context stage, product stage, process stage, and impact stage¹. As programmatic approaches funded through the State/Tribal and Campus programs differ, data collection activities have been tailored to the individual programmatic activities of each. The cross-site evaluation is designed to answer the following overarching questions:

• What types of prevention/intervention programs, services and products are used with youth identified as being at risk for suicidal behavior?

• What is the reach of program services, products, and strategies?

• To what extent does collaboration and integration influence referral mechanisms and service use?

• What is the impact of program services, products, and strategies on knowledge, process, and behavior?

The aim of the context stage is to gain an understanding of grantees’ program plans, such as target population, target region, service delivery mechanisms, service delivery setting, types of program activities to be funded, evaluation activities, existing data sources, and availability of data elements to support the cross-site evaluation. The product stage aims to describe the development and utilization of prevention strategies at each State/Tribal and Campus grantee site. The various prevention strategies may include awareness campaigns; outreach and awareness events; gatekeeper trainings; youth life skills development activities; policies and protocols for responding to youth at risk; means restriction strategies; screening programs; and enhanced services, such as early intervention, family support, and postsuicide intervention (postvention) services.

The process stage assesses progress on key activities related to implementation of grantee programs, such as the types of training conducted and roles of participants. All grantees are required to report aggregate training participant information for all trainings conducted as part of their suicide prevention programs. Data from State/Tribal grantees examine collaboration among different organizations/agencies involved in youth referral networks and how these networks change over time. Campus data include student exposure to suicide prevention activities on campus; awareness of resources; intended use of the skills, knowledge, and satisfaction with training; information learned through campus life skills and wellness activities; and suicidal thoughts. Finally, the impact stage examines the early impacts that suicide prevention programs have on individuals at risk for suicide. Existing information from States/Tribes on youth referred for services and service receipt from early identification activities and aggregate screening information for all youth screened is analyzed. For Campus grantees, administrative data related to the number students who seek mental health services, the type of services received, number of attempted or completed suicides among students, and the school retention rate are analyzed.

4. Clearance Request

SAMHSA is requesting approval for revisions to the previously-approved cross-site evaluation package (OMB No. 0930-0286; Expiration, January 31, 2017), now entitled the National Outcomes Evaluation. OMB clearance is requested for 3 years of data collection associated with the updated design, which represents SAMHSA’s desire to support the design, implementation, and dissemination of findings of a national impact evaluation of the GLS Suicide Prevention Program. The outcomes- and impacts-focused design will build on information collected through the cross-site evaluation four-stage approach, further the understanding of the impacts of the GLS Suicide Prevention Program, and meet the evaluation mandate outlined in the GLSMA. The evaluation now accounts for the differences in grant funding cycles (i.e., 5-year State/Tribal and 3-year Campus programs) and provides continuity and maximizes the use of previously collected data. By expanding on information gathered through the current evaluation on the process, products, context, and impacts of the GLS Suicide Prevention Program, the NOE will further the field of suicide prevention and mental health promotion.

2. Purpose and Use of Information Collected

The following is a description of the (a) National Outcomes Evaluation (introduction, current findings, data collection activities/methods, and major study components); (b) revisions to the previously-approved package; and (c) uses of the information collected.

1. National Outcomes Evaluation Overview

1. Introduction

The NOE proposed is a redesign of the currently-approved cross-site evaluation of the GLS Suicide Prevention Program (OMB No. 0930-0286; Expiration, January 31, 2017), first implemented in 2005. Building on findings to date, the NOE will focus on new priority areas of inquiry important to SAMHSA, Congress, and other suicide prevention stakeholders. The evaluation aligns with SAMHSA’s primary aim to assess the impact of GLS Suicide Prevention Program activities at reducing suicide attempts and deaths by suicide. The NOE will allow SAMHSA to expand the evidence base for suicide prevention; address factors contributing to suicide deaths and attempts; and establish standards for developing, implementing, and evaluating suicide prevention programs.

The NOE incorporates State/Tribal and Campus Program foci within and across programs and considers allowable activities, variation in partnerships and provider networks/infrastructure, program settings and populations being served, the range of program implementation plans and goals, existing data systems, and grant infrastructures to support implementation, and evaluation participation. The NOE also accounts for the differences in grant funding cycles across the two programs (i.e., 5-year State/Tribal and 3-year Campus programs) and maximizes the use of collected data. Combined with experimental and quasi-experimental methodologies and special analysis (described in Section A.16.c), the NOE will assess the impact of GLS programs on distal outcomes (i.e., suicide attempts, deaths, and hospital readmissions).

The NOE comprises three distinct, but interconnected core studies—Training, Continuity of Care (COC), and Suicide Safer Environment (SSE). The Training and SSE studies also have “enhanced” study components. Core study data align with required program activities across the State/Tribal and Campus programs and provide continuity with and utility of data previously collected (implementation and proximal outcomes). Enhanced components use experimental and quasi-experimental methods (randomized controlled trial [RCT] and retrospective cohort study designs) that provide an opportunity to study particular effectiveness and impact questions without undue burden on grantees and youth. This focus on impact, aligned with GLS program activities, will advance our understanding of what works, why, and under what conditions. Exhibit 1 below illustrates the major components of the NOE.

Exhibit 1. GLS Suicide Prevention Program National Outcomes Evaluation Components

The NOE aims to answer three overarching evaluation questions (EQs). By design, each of the three studies seeks to answer an overarching question and associated subquestions. Data collected through the RNS, TUP-S Campus, and SMSS will be combined with previously-collected data for those cohorts and analyzed as part of the four-stage evaluation design.

EQ-1: Training Study

• Are certain training approaches effective in building capacity to increase youth identification (when compared with more basic trainings)?

EQ-2: Continuity of Care Study

• Are GLS prevention activities effective in developing continuity of care from identification, to referral of at-risk youth, to the provision of needed services? Are early identification and referral practices associated with service receipt and follow-up treatment adherence compared with those not retained in services?

EQ-3: Suicide Safer Environment Study

• Does the provision of services by GLS behavioral health provider networks implementing a zero suicide framework reduce suicide attempts, hospitalization, and associated costs (return on investment) compared with non-GLS behavioral health providers?

2. Current Findings

Since 2005, the cross-site evaluation of the GLS Suicide Prevention Program has gathered evaluation data on GLS grantee programs, resulting in the greatest repository of suicide prevention evaluation information available. The cross-site evaluation has provided continuous documentation of the context in which funded suicide prevention activities are being implemented; the utilization of products and services being generated and supported through grant funding; and the impact of grant funding on identification, referral, and follow-up activities. Through participation in evaluation, GLS grantees are generating data regarding the nature and extent of suicide prevention activities across the United States, and have provided important data regarding the impact of programs in terms of numbers of individuals affected (e.g., screened, trained) and proximal outcomes of efforts (e.g., increased knowledge or awareness, numbers of youth at risk referred for services). Moving forward, the capacity and infrastructure of the GLS Suicide Prevention Program need to be elucidated to facilitate the early identification, referral, and follow-up of youth at risk (Heilbron, Goldston, Walrath, Rodi, & McKeon, 2013; Rodi et al., 2012).

Gatekeeper training has been identified as a critical element in suicide prevention efforts (Isaac et al., 2009). Over 747,108 individuals have participated in trainings and educational seminars sponsored by the GLS Suicide Prevention Program since 2006 (ICF International, 2014). These trainings have been found to increase knowledge of suicide intervention, skills, attitudes, and intention to help someone at risk for suicide, including school counselors and teachers (King & Smith, 2000; Reis & Cornell, 2008; Wyman, 2008); juvenile justice and child welfare staff members (Keller et al., 2009); those working with veterans (Matthieu, Cross, Batres, Flora, & Knox, 2008); and others (Isaac et al., 2009). Findings from the evaluation indicate that GLS-trained gatekeepers are identifying youth at risk across service settings, and those youth are being referred for services without regard for race, gender, or the settings in which they are identified (Rodi et al., 2012). The evaluation has also provided initial findings that indicate a positive collective impact of GLS Suicide Prevention-sponsored suicide prevention trainings on subsequent identification behavior of trainees (Condron, Godoy-Garraza, Walrath, McKeon, & Heilbron, 2014) and establishes the effect of GLS Suicide Prevention Program trainings on youth suicide attempts and suicide mortality; findings indicate that counties where GLS trainings were implemented had lower suicide rates in the year following training events compared to similar counties that did not have GLS trainings (Walrath, Godoy-Garraza, Reid, Goldston, & McKeon, in press; SAMHSA, 2013b).

Further, after identification and referral, best practices call for tracking and monitoring of youth into follow-up services to ensure service receipt and prevent youth from ‘falling through the cracks’ after identification. To date, evaluation have demonstrated that the majority of youth identified through either a screening (75%) or a gatekeeper (88%) received a mental health referral. Nearly all youth who were identified through gatekeepers (94%) received a mental health service within 3 months of the referral. However, while early findings of the GLS Suicide Prevention Program have demonstrated that 65% of referred youth received a service within 3 months of the referral, approximately 35% did not (SAMHSA, 2013). Learning more about this pathway of care, as well as the factors that support follow-up care and treatment adherence, will be important to guide future policies and practices for supporting youth identified both through gatekeeper identifications and screenings.

Collectively, this information has been used to help guide the field of suicide prevention across the nation's communities; document the GLS Suicide Prevention Programs; and contribute to findings on the relationship of training length and identification of youth at risk for suicide and overall reductions in suicide death during the year following trainings. For example, ICF has recently shown that, for participants typically interacting with youth in school settings, a larger number of identifications were associated with participation in longer gatekeeper training when compared with shorter trainings 3 months after the activity (Condron et al., 2014). However, there has been little use of experimental approaches (e.g., randomization and controls) that would allow inferences to be drawn about the efficacy or effectiveness of suicide prevention programs in affected versus non-affected communities. In addition, it has been inappropriate to draw definitive inferences about the comparative effectiveness of suicide prevention approaches (e.g., one method of suicide screening or gatekeeper training versus another) used in different GLS funded sites.

The updated design ensures that the methods selected maximize the opportunity for high-quality data collection addressing impact, while working to overcome common challenges. In reframing the scope of the evaluation, the design will allow for this, as well as the use of rigorous methodological approaches that provide a means of addressing current questions in this next stage. Moving forward, it will be important to determine, among other priorities, how to improve the adoption, fidelity of implementation, and sustainability of effective suicide prevention programs, with attention to efficient ways of training various types of gatekeepers and providers.

3. Data Collection Instruments and Methods

Approval is being requested for 11 data collection activities that compose the NOE. Because GLS program foci differ by type of grantee, some instruments apply only to State/Tribal or Campus programs. Of the 11 instruments, 2 will be implemented with State/Tribal and Campus grantees (the versions are slightly tailored by grantee type), 6 are specific to State/Tribal grantees, and 3 pertain to Campus grantees only. A description of the instruments, instrument status (revised or new), and applicable grantee type(s) is below. Further detail on data elements, respondents, and methods of administration can be found in Attachment B; changes to existing instruments are described in Exhibit 5, Revisions to the Evaluation, in Section A.2.b. A full list of attachments is located at the end of the Supporting Statement.

Instrument Key

Below is list of acronyms and names that apply to NOE instruments.

Attachment Acronym Name

C PSI Prevention Strategies Inventory

D TASP Training Activity Summary Page

E, F TUP-S Training Utilization and Preservation Survey

G EIRF-I Early Intervention Referral and Follow-up Individual Form

H EIRF-S EIRF Screening Form

I SBHF Student Behavioral Health Form

J BHPS Behavioral Health Provider Survey

K RNS Referral Network Survey

L SMSS Short Message Service Survey

State/Tribal and Campus Instruments

Two instruments are applicable to all State/Tribal and Campus grantees: the revised PSI and TASP.

PSI (Revised): the PSI is a Web-based survey that captures all State/Tribal and Campus program prevention strategies and products. Data include strategy types and products distributed, target populations, and expenditures across major categories (e.g., outreach and awareness, gatekeeper training, screening programs, etc.). Each major strategy includes sub-strategies, enabling grantees to specify and provide details about the strategy or product and targeted populations. The PSI is completed by grantee staff each quarter. PSI data will inform the Training, COC, and SSE Study core components.

TASP (Revised): the TASP is a Web-based survey collecting aggregate-level data from all State/Tribal and Campus grantees. Data include information about the number and role of participants, type of training offered, training setting, and training location ZIP code (for use in analysis of training impact). The TASP also assesses intended outcomes, as well as the number of online trainings completed, train-the-trainer events held, and booster trainings that follow the initial training. A TASP is completed by grantee program staff within 2 weeks of each training activity. TASP data will inform the Training Study core.

State/Tribal Instruments

State/Tribal grantees will participate in the revised TUP-S, EIRF-I, and EIRF-S; new BHPS; and new TUP-S baseline and 12-month administrations. Cohort 8 State/Tribal grantees also will participate in the previously-approved RNS after which the instrument will be discontinued.

TUP-S (Revised/New): the TUP-S is a survey conducted with training participants from GLS State/Tribal grantees. The TUP-S has 3- and 6-month versions; as part of the Training Study RCT, two new versions (baseline [BL] and 12 months) are being added. The TUP-S systematically measures gatekeeper behavior. Data include training participant (1) self-efficacy, awareness, and education efforts; (2) receipt and utilization of materials and tools to support skills; (3) experience with role-play or behavioral rehearsal; (4) previous suicide prevention trainings; (5) previous contact and quality of relationships with youth and students; and one of the most critical outcomes, (6) suicide identification and referral behavior. Each version refers the participant to a specific time period. As the initial follow-up survey, the 3-month version also requests information about the training received. All versions are administered using CATI technology. TUP-S data will inform the core COC and Training Studies, as well as the Training Study RCT.

• TUP-S Core (3 and 6 months): for the core Training Study, the TUP-S will be administered at 3 months and 6 months after training with a random sample of participants who consent to be contacted. State/Tribal and Campus grantee staff and training facilitators will collect consent to contact prior to training via hardcopy form or by sharing a link to an online form (mobile device-compatible). Brief items have been added to the consent to contact form to establish participant history of identification and referral behaviors. Respondents at 3 months will be asked to consent to be contacted for the 6-month survey.

• TUP-S RCT (BL, 3, 6, and 12 months): as part of the Training Study RCT, the TUP-S will be conducted with participants from a subset of 10 State/Tribal grantees randomly assigned to different training activities. Grantee staff will collect consent-to-contact at registration for upcoming training events via hardcopy form or by sharing a link to an online form (also mobile device-compatible). Prospective participants will complete the baseline survey prior to training and follow-ups at 3 months, 6 months, and 12 months thereafter.

Note: The core and RCT Training Study components both will use the TUP-S 3- and 6-month versions. To distinguish between core and RCT data collection, TUP-S RCT refers to any version of the survey conducted as part of the RCT. For example, the 3-month TUP-S and TUP-S RCT versions are the same instrument. However, “RCT” denotes data collection specific to the RCT.

EIRF-I (Revised): the Web-based EIRF-I gathers existing data from all State/Tribal grantees on each at-risk youth identified as a result of the GLS program (via trained gatekeepers or screenings). Data include information on service referrals, service types, and services received across the 3-month period after identification; no personal identifiers are requested. Information sources include local systems that gather identification and referral data, including existing data extractions from electronic health records or forms. Grantee program staff enter EIRF-I data on an ongoing basis. EIRF-I data will inform the core Training and COC Studies.

EIRF-S (Revised): the Web-based EIRF-S gathers aggregate information about all State/Tribal screening activities sponsored by the GLS program. Data include aggregate information on the number of youth screened for suicide risk through the GLS program, the number screening positive, and the number confirmed to be at risk after initial positive screening. Grantee program staff complete forms once per each implementation of a screening tool in a group setting, once per month for clinical screenings, and once per month for one-on-one screenings. EIRF-S data will inform the core COC Study.

BHPS (New): the BHPS is a new Web-based survey that gathers a central set of characteristics about behavioral health providers partnering with State/Tribal grantees. Data will include information about referrals for at-risk youth, SSE care practices implemented, and client outcomes (number of suicide attempts and deaths). The survey will be administered to providers from partner organizations annually. A total of 1-10 behavioral health partners from each State/Trial grantee will participate. BHPS data will inform the core and enhanced SSE Studies.

RNS (No Changes): the RNS is Web-based survey of organizations in State/Tribal referral networks that identify, refer, or serve at risk youth. The RNS examines how collaboration and integration are used to share and transfer knowledge, resources, and technology among State/Tribal grantees and stakeholders. Data include information on (1) how networks influence referral mechanisms and service ability and (2) policies and protocols regarding follow-up for youth who have attempted or are at risk for suicide. Grantee program staff provide contact information for up to 5 organizations in the referral network. The RNS will be administered in OMB Year 1 with Cohort 8 State/Tribal grantees to fulfill data collection requirements for that cohort.

Campus Instruments

All Campus grantees will participate in the revised SBHF (formerly MIS). Campus Cohorts 7 and 8 will participate in the previously-approved TUP-S Campus; Cohort 7 campuses also will participate in the previously-approved SMSS.

TUP-S Campus (No Changes): the TUP-S Campus is a telephone survey conducted with training participants from GLS Campus grantees. The TUP-S systematically measures gatekeeper behavior. Data include self-efficacy, awareness and education efforts; suicide identification behavior; demographic information about individuals identified at risk; subsequent referrals and/or supports provided by the trainee; and any available information about services accessed by the at-risk individual. Grantee staff and training facilitators will collect consent to contact prior to training via hardcopy form. The TUP-S Campus will be administered to a random sample of 500 training participants from Cohort 7 and 8 Campus grantees in year 1 of the OMB data collection period to fulfill data collection requirements for the cohorts.

SBHF/MIS (Revised): the SBHF (formerly the MIS) is a Web-based survey that gathers existing data in aggregate from all Campus grantees about behavioral health services and student suicidality and mortality. Data include information on (1) nature/type of available campus behavioral health services; (2) number of students receiving behavioral health services annually; (3) the nature and type of services received; (4) screenings used through behavioral health services; and (5) populations screened (universal or selective), number of students identified as at-risk through screenings, and services provided for students identified as at-risk. SBHF data will inform the core COC Study and the core SSE Study.

SMSS (No Changes): the SMSS is a 4-question survey of students from Campus grantees that examines patterns of suicide risk and exposure to prevention efforts. Data include information about student exposure to campus suicide prevention program efforts and history of suicidal ideation and attempts. The SMSS will be administered to 100 students from each Cohort 7 campus in OMB Year 1 to fulfill data collection requirements for the cohort.

1. Major Study Components

The NOE consists of 3 interconnected and simultaneously-launched studies—Training, Continuity of Care (COC), and Suicide Safer Environment (SSE). A description of each study, the study purpose, and associated data collection activities follows.

Training Study

Since 2006, more than 747,108 individuals have participated in GLS Suicide Prevention Program sponsored trainings and educational seminars (ICF International, 2014). In Fiscal Year (FY) 2014, over 90% of GLS grantees reported implementing gatekeeper training as a core strategy of their suicide prevention programs and allocated over 25% of their budgets toward the activity. Further, in the same FY, Campus, State, and Tribal grantees implemented more than 25,000 training activities as part of their GLS Suicide Prevention Programs. Given the emphasis on training by grantees, the NOE will continue to assess training activities and their impact.

Goal 7 of the NSSP is to “provide training to community and clinical service providers on the prevention of suicide and related behaviors.” Within this, Objective 7.1 is to “provide training on suicide prevention to community groups that have a role in the prevention of suicide and related behaviors” (HHS, 2012). As such, training of individuals who are in contact with youth at risk for suicide is a commonly implemented and important prevention effort. While previous studies of gatekeeper training effectiveness have found that trainings effect more immediate outcomes (Isaac et al., 2009), less is known about the effect of gatekeeper trainings on intermediate outcomes (identifications and referrals) and distal outcomes (attempts and deaths). Results from a randomized controlled trial (RCT) of a gatekeeper training (Question, Persuade and Refer [QPR]) found the evidence of the effect of gatekeeper training on identifications and referrals inconclusive, except among gatekeepers who were already communicating with youth (Wyman et al., 2008). Additionally, although gatekeeper training impacts knowledge and awareness, many trainees’ skills decreased over time. Cross et al. (2011) found that role-play practice and feedback during training improved retention of gatekeeper skills, especially the ability to ask an individual directly about suicide and the ability to communicate with someone in distress.

Although studies have demonstrated benefits to gatekeeper training, limitations include not utilizing randomized groups. The overarching purpose of the Training Study is to increase our understanding of the impact of suicide gatekeeper training on participant identification and referral behaviors; barriers and facilitators to using the skills learned; factors that may mediate this impact (e.g., different relationships and interactions with youth before the training); and the cost of implementing trainings. The study includes core questions about trainings implemented and their proximal outcomes, as well as an RCT to assess the effectiveness of key training components on promoting early identification. The core study component will help create an understanding of the reach of GLS-sponsored suicide prevention trainings, in addition to examining the characteristics of the trainings, trainees, and settings that predict differences in participant knowledge, skills, and prevention behavior following the trainings. The enhanced RCT study component assesses the impact of specific training enhancements, such the use of active learning strategies and implementation of booster trainings, on increasing identification and referral behaviors. A logic model for the Training Study can be found in Attachment M.

Data from five instruments will inform the Training Study in its aim to answer EQ-1, as well as subquestions from the core and RCT study components. An overview of the components, related study questions, and associated instruments is presented in Exhibit 2.

Exhibit 2. Training Study Components, Questions, and Instruments

EQ-1: What types of training and training modes are comparatively effective in building capacity to increase youth identification?	PSI	TASP	TUP-S	TUP-S RCT	EIRF-I
Core
1.1: What type of training curricula are grantees implementing?	x	x
1.2: Which populations are being trained?	x	x	x
1.3: What are the expected outcomes associated with training participation?		x
1.4: What factors related to the training (e.g., use of active learning strategies), trainee (e.g., ‘natural gatekeepers’), and setting (e.g., ‘threshold effects’) predict differences in gatekeeper behavior following participation?		x	x
Randomized Control Trial
1.5: How are trainee knowledge, skills, awareness, and self-efficacy maintained over time?			x	x
1.6: Does the use of active learning strategies increase effectiveness of gatekeeper trainings (particularly in terms of promoting identification of youth at risk)?				x
1.7: Does the use of booster interventions following training participation increase effectiveness of gatekeeper trainings (particularly in terms of promoting identification of youth at risk)?				x
1.8: Do training components (role-play, booster) impact particular segments of trainees differently?			x	x
1.8.1: How does the trainee relationship and amount of time spent with youth impact identification and referral of youth at risk for suicide?			x	x	x
1.8.2: How does previous experience with suicide (via training or contact with suicidal individual) impact identification of youth at risk for suicide?			x	x
1.9: Are there variations on comparative effectiveness according to the setting characteristics in which trainees interact with youth?			x	x

Training Study Core

The core study data collection activities help GLS grantees track aggregate training information, as well as participant post-training experiences, identifications, and referrals. These data are also used by grantees on a local level to respond to SAMHSA program-level indicators (through the Common Data Platform).

All State/Tribal and Campus grantee programs implementing suicide prevention training will contribute data to the core study by tracking background information on implemented trainings via the TASP. All grantees are required to complete the PSI, which will track training-related prevention strategies implemented as part of their programs, as well as the training-related expenditures. The core study also will recruit and follow a random sample of trainees from all grantees. Participants who agree to be contacted for a follow-up survey will answer brief questions about their experience identifying and referring youth at risk prior to participant. The TUP-S follow-up survey will then assess training participant knowledge, awareness, and identification and referral behavior patterns at two points in time, specifically 3 and 6 months after the training.

All State/Tribal grantees are required to participate in the EIRF-I, which will provide additional context for the self-report data shared by trainees. In addition, EIRF data will be used to measure the broader impact of training on the trainee communities (e.g., community based organizations and school, child welfare, or/or juvenile justice agencies). For this aspect, information from the TASP (e.g., percentage trained within same setting) will be reviewed to understand more about the contextual effects of training and connecting training to a broader system or community effects.

Training Study RCT

The enhanced RCT study will be implemented to understand how the use active training strategies and booster trainings affect training participant knowledge, skills, and identification and referral behaviors over time. The RCT uses a factorial randomized controlled design (Montgomery, Peters, & Little, 2003) to examine the impacts of four different training interventions:

• Gatekeeper training

• Gatekeeper training + role-play

• Gatekeeper training + booster

• Gatekeeper training + role-play + booster training.

A sample of 10 State/Tribal grantees will be recruited to implement the four training interventions and help recruit training participants locally for the RCT. Participant grantees will implement gatekeeper trainings that are intended for adults over the age of 18; relatively brief (about 1 to 3 hours in duration); emphasize the outcomes of interest for the study; are intended to share information about identification, assessment, and referral of suicidal individuals; and emphasize questioning individuals who may be at risk for suicide (QPR: Quinnett, 1995). All RCT training participants who consent to be contacted will be eligible to participate in the study, specifically the TUP-S RCT. The TUP-S RCT will be administered at four points in time: baseline (pre-training) and 3, 6, and 12 months after the training. Participants will receive a consent-to-contact form at registration for the training that asks brief questions about them and their history of identification and referrals. A total of 1,332 training participants will be recruited for the RCT, with 333 participants randomly assigned to each training intervention.

Continuity of Care (COC) Study

The COC Study will be used to understand the process of identifications and referrals supported by the GLS Suicide Prevention Program, as well as what happens to youth/students who are identified as a result of the program. The study will assess the early identification, referral, and follow-up practices of GLS grantees, as well as the results of these efforts. The study examines the extent to which youth, identified as a result of the GLS Suicide Prevention Program, receive follow-up mental health services and other supports after their identification. A detailed logic model for the COC is located in Attachment N.

An integral component of GLS State/Tribal programs is building capacity and infrastructure to identify and refer at-risk youth through the implementation of gatekeeper trainings and suicide risk screenings. A best practice for both mechanisms is that a system be in place to monitor and ensure that follow-up services are received after referral. Additional recommendations state that gatekeeper training must include post-identification protocols, community-specific suicide prevention resources, and supports that are available where the trainee works and/or lives—all of which demand that adequate supports/services are in fact, available to at-risk youth. Recommendations for screening include developing response protocols for youth perceived at risk, including imminent risk, to ensure the receipt of immediate guidance and referral—also demanding the availability of adequate services in the event that an at-risk youth is identified. Further recommendations strongly encourage a directory of available community resources for behavioral health care that can be used for referral after identification.

For Campus program grantees, developing the infrastructure to deliver effective mental health and substance abuse prevention, treatment, and recovery support services is critical—whether or not the campus itself provides comprehensive behavioral health services. Campuses not providing these services must create a network infrastructure linking the institution to health care providers from the broader community, specifically providers who can provide culturally and linguistically appropriate treatment and recovery support services. Campus grantees must also have essential capacities in place—crisis response protocols, information and referral to adequate on- and off-campus behavioral health services, and trained staff—before adding new efforts so that demand for services does not outpace capacity.

Eight instruments will inform the COC Study in its aim to answer EQ-2 and 7 subquestions. Exhibit 3 presents an overview of study components, questions, and instruments.

Exhibit 3. COC Study Components, Questions, and Instruments

EQ-2: Are GLS prevention activities effective in developing continuity of care from identification, to referral of youth at risk for suicide, to the provision of needed services?	PSI	EIRF-I	EIRF-S	SBHF	BHPS
2.1: What proportion of youth identified by GLS grantees receive follow-up support?		x
2.2: Do youth characteristics predict follow-up support and adherence to follow-up care (e.g., gender, race-ethnicity) or setting of identification?		x
2.3: Does the proportion of follow-up care increase over the duration of GLS program activities?		x
2.4: What are the practices and supports used by the grantees to ensure that identified youth receive referral and follow-up?	x	x			x
2.5: What are the gaps in support for youth identified as at risk for suicide?		x
2.6: What are the patterns of identification, referral, and follow-up for youth identified as at risk for suicide?		x	x	x
2.7: What are the follow-up services received by at-risk youth?		x		x

COC Study Core

The COC core comprises primary and secondary data collection activities—surveys, inventories, and existing data abstractions—to document the GLS-sponsored prevention activities that support and contribute to the early identification, referral, and follow-up of students and youth at risk for suicide. All grantees will contribute data to the COC core; however, some data collection activities apply only to Campus or State/Tribal grantees.

State/Tribal and Campus grantees will contribute to the COC through the PSI and TUP-S. The PSI gathers the prevention strategies for the early identification, referral, and follow-up of at-risk youth and students; how grantees track and monitor at-risk youth identified through screenings; and the follow-up protocols for each screening activity (e.g., the protocols and tracking tools used to ensure that youth referred for services are followed up with and get to adequate mental health or other support referral sources). TUP-S data will provide context to the practices and protocols of trained gatekeepers to ensure that youth receive follow-up services; what roles and responsibilities the trained gatekeeper has (or was trained in) relative to tracking or monitoring follow-up on any at-risk youth identified; and what they did to ensure access to mental health services (or other support services) after their identification of at-risk youth.

Separately, all State/Tribal grantees will complete the EIRF-I and EIRF-S to understand the practices and outcomes (receipt of follow-up services) associated with early identification and referral of youth. The EIRF–I will gather individual-level (de-identified) data on early identification, referral, and follow-up information for youth identified through the program (i.e., through GLS-trained gatekeepers or through GLS-sponsored screenings), while the EIRF–S collects aggregate data on the number of youth screened for suicide risk, the number screening positive, and the number confirmed to be at risk after initial positive screening.

For all Campus program grantees, the annual SBHF data abstraction and submission will provide the number of students identified as being at risk, including students that self-present or are referred for campus counseling or psychological services. Additional information will include aggregate data on GLS-sponsored screenings, populations screened (universal or selective), the resulting number of at-risk identifications and services provided, and the number of student suicide attempts and completions.

Suicide Safer Environment (SSE) Study

The SSE Study assesses the extent to which grantees and partnering providers are implementing Goal 8 and 9 practices of the NSSP (HHS, 2012), which move away from more primary prevention activities, such as wellness programs, community awareness, and improving community readiness, and emphasize integrating suicide prevention as a core component of health care and implementing effective clinical and professional practices to assess and treat at-risk individuals. The focus is on treatment and support services for individuals at high risk of suicide and whether these practices lead to the long-term outcomes of reductions in suicide deaths and attempts. Taken together, the goals include 15 primary objectives summarized by the primary components of the Zero Suicide framework that can lead to suicide safer health care environments:

• Creating a Zero Suicide culture committed to dramatically reducing suicide among people under care

• Systematically identifying and assessing suicide risk level among people at risk, using credible screening tools followed by clinical assessment

• Ensuring every person has a pathway to care that is both timely and adequate to meet their needs

• Developing a competent, confident, and caring workforce

• Using effective evidence-based care including collaborative safety planning, restriction of lethal means, and effective treatment of suicidality

• Continuing contact and support, especially after acute care

• Applying a data-driven quality improvement approach to inform system changes

State/Tribal grantees funded in FY 2014 or later include a focus on implementing Goals 8 and 9 and to reduce rates of suicidal ideation, suicide attempts, and suicide deaths in their communities. Through partnerships with behavioral health providers, grantees are expected to apply key elements of the Zero Suicide Toolkit, developed by the Clinical Care and Intervention Task Force of the National Action Alliance for Suicide Prevention (Action Alliance). This toolkit identifies essential dimensions of suicide prevention for health care systems, including health care plans or care organizations serving a defined population of consumers (e.g., behavioral health programs, integrated delivery systems, and comprehensive primary care programs).

The SSE core is focused on the development of a suicide safe environment using a Zero Suicide framework for behavioral health provider networks or campus health services, and its association with proximal and distal outcomes, including suicide attempts and suicide mortality. The enhanced study uses a claims-based retrospective cohort design to examine the effects of these networks/provider services by comparing outcomes (e.g., attempts, mortality, hospitalizations) of youth served in enhanced suicide safer environment contexts with those receiving usual care (providers not implementing a suicide safer environment framework) using extant Medicaid claims data (MAX). A summary logic model for the SSE can be found in Attachment O.

The aim of the SSE Study is to answer EQ-3, along with 6 subquestions from the core and cohort study components. The study assesses the extent and degree of success with implementing toolkit practices among State/Tribal partnering behavioral health providers. Related practices implemented within campus health services also will be assessed. Together, SSE data sources include: PSI, BHPS, SBHF, and MAX. An overview of the SSE components, subquestions, and associated instruments is presented in Exhibit 4.

Exhibit 4. SSE Study Components, Questions, and Instruments

EQ-3. To what extent are grantees and associated providers implementing suicide safer environment frameworks? Does this framework implementation lead to decreases in suicide attempts and completions?	PSI	SBHF	BHPS	Extant Data
Core
3.1: What specific elements and activities of a suicide safer environment framework (particularly Goals 8 and 9 of the National Strategy are being implemented by behavioral health providers involved in the GLS State/Tribal Program?		x	x
3.2: What suicide safer environment care activities are being implemented by campus health services?	x	x
3.3: Are access and utilization of campus behavioral health services higher for students on campuses that integrate clinical screenings or suicide assessments into campus primary and behavioral health care?	x	x
3.4: Are suicide ideation, attempts, and completions lower for students on campuses that are more fully implementing the suicide safer environment framework activities?	x	x		x
Retrospective Cohort Study (Medicaid Claims Analysis)
3.5: Are suicide attempts reduced for youth receiving services by GLS behavioral health providers compared to non-GLS providers?			x	x
3.6: Are hospital readmissions and in-hospital deaths by suicide lower for youth (at risk for suicide) who use services from providers that have implemented National Strategy Goals 8 and 9 activities?			x	x

SSE Study Core

The SSE core study uses primary and secondary data collections—student surveys, provider surveys, and behavioral health services data extraction activities—to document the nature, extent, and changes in SSE practices over time. Together, the data collection activities also address the long-term outcomes of reductions in suicide attempts and deaths and associations with SSE care implementation. Across the core SSE data collection activities, there is no site selection required, as the activities apply to all relevant Campus or State/Tribal grantees.

Across State/Tribal grantees, the bulk of the suicide safer care activities will occur within partnering behavioral health provider organization or other providers in associated referral networks who identify and refer at-risk youth to health care settings. State/Tribal core activities will gather implementation and outcome information from provider and referral organizations. Grantee partnering providers will provide information about SSE care practice implementation and client outcomes (number of suicide attempts and deaths) through the BHPS.

Within the Campus grantee setting, the SSE Study applies primarily to campus behavioral health provision—for example, looking at suicide safer care practice implementation within campus counseling centers or suicide screening and referral practices within campus primary health care settings. All Campus grantees will complete the SBHF data abstraction and submission which gathers existing de-identified data about the types and numbers of behavioral health services provided on campus, the number of youth screened for suicide risk, referrals to services off campus, student suicide attempts and completions, and the number of follow-ups with at-risk students identified and/or those who attempted suicide. Information about screening, referring, and assessing suicide risk procedures, as well as service provision help to document the extent of suicide safer practices implemented within campus health services.

SSE Cohort Study (MAX Analysis)

No primary data are directly collected for the enhanced SSE. Rather, the study uses Medicaid claims data to examine the effects of provider services by comparing outcomes (attempts, mortality, hospitalizations) of youth served in enhanced suicide safer environment contexts with youth receiving usual care (providers not implementing a suicide safer environment framework). The study will look in-depth at the outcomes of clients served by GLS-partnering providers and draw on extant claims data as the main outcomes source for GLS-partnering providers and providers not associated with GLS grantees (as a control in analyses).

Given the SSE focus on health care settings, it is critical to develop a full understanding of the SSE care practices undertaken by GLS partnering providers, as well as relationship of care practices with long-term outcomes (reductions in suicide attempts and deaths), particularly nonfatal suicide attempts requiring medical attention. To effectively measure suicide attempts and deaths among client populations served by GLS partnering providers, a control group must be used. Thus, the enhanced study component draws on extant Medicaid claims data (MAX)—specifically, patients with Medicaid coverage—to compare outcomes across two cohorts:

• 1,000 youth identified as at risk through the GLS program for the year following their contact with a GLS behavioral health provider (test group)

• 1,000 youth experiencing similar circumstances for the year following their contact with non-GLS providers (control group)

The MAX claims data can be used to (1) identify the population at risk of suicide who reach the health service system (through diagnosis and service use), and (2) obtain pertinent patient-level outcome measures (injury requiring medical attention, hospital admission, in-hospital deaths). Longitudinal information on the same patient is available across multiple health care providers when the patient remains with the same insurance provider. MAX can be linked to the provider-level information collected through the BHPS using provider identifiers. All States/Tribes with available claims data and BHPS data will be included in the enhanced claims analysis. Because the MAX data are available for all States for a single access fee, there will be flexibility around cohort sizes if necessary.

2. Revisions

Exhibit 5 contains a summary of revisions to the previously-approved cross-site evaluation package and the rationale behind each of the changes.

Exhibit 5. Revisions to the Evaluation

Revision	Rationale
Title	The cross-site evaluation will be renamed the National Outcomes Evaluation (NOE), reflecting SAMHSA’s desire to focus on outcomes of the GLS program.
Burden Calculation Period	Through the most recent OMB clearance for the cross-site evaluation was requested and approved for 3 years of data collection through January 2017. Respondent burden for this revision request is calculated for the next 3 years of data collection, from January 2016 to January 2019.
Grantee Participants	The number of grantees for which burden is calculated is 140 (49 State/Tribal grantees and 91 Campus grantees), representing the total number of grantees that will be funded across the 3-year OMB period. Campus grantee numbers and funding years are as follows: 22 (FY2013), 15 (FY2014), 22 (FY2015), 17 (expected FY2016), and 15 (expected FY2017). All campus grantees have 3-year grant cycles. State/Tribal grantee numbers and funding years include: 7 (FY 2013), 26 (FY 2014), 12 (FY 2015), and 4 (expected FY 2016). None are expected to be funded in FY 2017. Grantees funded in FY 2013 have 3-year funding cycles; grantees funded in FY 2014 and beyond have 5-year grant cycles.
Evaluation Questions	The NOE aims to answer 3 primary evaluation questions and multiple subquestions that assess the implementation, outcomes, and impacts associated with the GLS program—a change in focus and number of questions from the previous package.
Design	The evaluation design has been updated to reflect SAMHSA’s desire to assess the implementation, outcomes, and impacts of the GLS Suicide Prevention Program. Building on information collected through prior GLS evaluations, the NOE will contribute additional evidence on: (1) suicide prevention training effectiveness, (2) the impact of early identification and referral on subsequent care follow-up and adherence, and (3) the impact of GLS behavioral health providers implementing the SSE framework with at-risk youth. The NOE consists of three core studies—Training, COC, and SSE. The Training and SSE Studies also have enhanced study components (RCT and retrospective cohort studies, respectively). Core study components align with grantee program requirements. Enhanced studies utilize experimental and quasi-experimental design and methods to assess program impacts. The design also incorporates both the State/Tribal and Campus programs’ foci within and across programs and communities and accounts for the difference in funding cycles between the State/Tribal and Campus programs.
Instrument Removals	Due to the fulfillment of data collection requirements, 6 previously-approved instruments will be removed: Coalition Profile (CP), Coalition Survey (CS), TUP-S Adolescent, Training Exit Survey (TES) Individual Forms, Life Skills Activities & Follow-up Interview (LAFI), and Student Awareness Intercept Survey (SAIS).
Instrument Continuations	To satisfy existing data collection requirements for the current design protocol, the RNS, TUP-S Campus, and SMSS will be administered with certain grantees during OMB Year 1. Each instrument has been previously approved and no changes are being made. These instruments are not a part of the new proposed evaluation studies and will be discontinued after existing protocols are completed. State/Tribal Cohort 8 grantees will participate in the RNS, completing their 2nd administration of the instrument. The RNS takes 40 minutes to complete. Campus Cohorts 7 and 8 will participate in the TUP-S Campus. The TUP-S Campus takes 10 minutes to complete. Cohort 7 campuses will participate in the SMSS, completing their 2nd administration of the instrument. The SMSS takes 5 minutes to complete.
Instrument Additions	Two new instruments will be incorporated into the NOE: TUP-S (baseline and 12-month RCT versions) and BHPS.
Instrument Revisions	Six previously-approved instruments will be revised as part of the NOE: the PSI, TASP, TUP-S, EIRF-I, EIRF-S, and SBHF/MIS.
PSI Revisions	The PSI has been updated to enhance the utility and accuracy of the data collected. Changes capture different strategies implemented and products distributed by grantee programs, the population of focus for each strategy, total GLS budget expenditures, and the percent of funds allocated by the activity type. Data from the PSI will address the core components of the Training Study, Continuity of Care Study, and Suicide Safer Environment Study. The PSI takes 45 minutes to complete.
TASP Revisions	New items on the TASP gather information about the use of behavioral rehearsal and/or role-play and resources provided at trainings—practices that have been found to improve retention of knowledge and skills posttraining. In addition, understanding how skills can be maintained over time with materials provided at trainings (e.g., video reminders, wallet cards, online and phone applications) is an area suggested for further study (Cross et al., 2011). It is estimated that project staff will spend 15 minutes completing TASPs quarterly.
TUP-S Revisions	The TUP-S has been improved to examine posttraining behaviors and utilization of skills by training participants—factors known to improve understanding of the comprehensive training process and the impact that training has on identifications, referrals, and service use. The survey now requests detailed information about training resources received, practice components, trainee participation in role play, and previous suicide prevention trainings attended. Experience intervening with a suicidal individual (from QPR evaluation tool), intended use of the training, and referral behaviors also will be collected. Other items will measure previous contact and quality of relationships with youth. Broad items about training others, the use/intended use of skills, and barriers/facilitators have been removed. The consent-to-contact form has been modified to add brief items about the trainee and previous identifications/referrals and takes 10 minutes to complete. Broad items about training others, the use/intended use of skills, and barriers/facilitators have been removed. The TUP-S will be administered at 3 and 6 months post-training to a random sample of training participants via CATI (2000 ST TUP-S 3-mo/600 ST TUP-S 6-mo per year). The TUP-S takes 30 minutes to complete at 3 months and 25 minutes to complete at 6 months.
TUP-S RCT New Versions	Two new versions of the TUP-S (BL and 12 months) are being added for use in the Training Study RCT and take 25 minutes to complete. The TUP-S RCT refers to all TUP-S versions administered as part of the Training Study RCT. The RCT collects TUP-S data at baseline (pre-training) and 3, 6, and 12 months after training. Because the surveys are conducted at different times, each version refers the participant to a specific time period. All State/Tribal trainees participating in the RCT who consent to be contacted will be surveyed until the desired sample size of 1332 respondents is achieved. The consent-to-contact form will describe the RCT and the 4 assessment periods.
EIRF-I Revisions	The EIRF-I has been improved to gather initial follow-up information about youth identified as being at risk as a result of the GLS program (whether or not a service was received after referral). EIRF-I data elements have been expanded to include screening practices, screening tools, and screening results of youth identified as at-risk for suicide. Response options have been expanded/refined: setting/source of identification, mental health and non-mental health referral locations, and services received. Tribal-specific data elements have been added and the sources of information used has been removed. It is estimated that project staff will spend 45 minutes completing the EIRF-I each quarter.
EIRF-S Revisions	Data elements have been added to indicate whether screenings were performed at the individual- or group-level. New response options have been added under “screening tool” and “false positive” has been removed as an option. It is estimated that project staff will spend 45 minutes completing the EIRF-S each quarter.
BHPS New Instrument	The BHPS is a new data collection activity and the first to specifically target behavioral health providers partnering with GLS grantees. Data will include information about referrals for at-risk youth, SSE care practices implemented, and client outcomes (number of suicide attempts and deaths). A total of 1-10 behavioral health partners from each State/Trial grantee will participate annually. The BHPS takes 45 minutes to complete.
SBHF/MIS Revisions	The SBHF (formerly entitled the MIS) has been expanded and renamed. The form has been enhanced to include referral and follow-up procedure questions (rather than simply counts); numbers screened, identified at risk, receiving suicide-specific services, referred, and receiving follow-up; and age and gender breakdowns of suicide attempts and deaths. Student enrollment/retention items have been removed; these will be obtained through the Integrated Postsecondary Education Data System (IPEDS). The SBHF takes 40 minutes to complete. It will require closer involvement with campus behavioral health/health providers to gather data on procedural questions and screenings, risk assessment, services, referrals, and follow-ups.
Special Analyses	Three types of special analyses—Ecological Impact, Outcomes Modeling, and Unintended Consequences—are proposed to address effectiveness and impact questions and use primary data collected by ICF for previous evaluations of the GLS Suicide Prevention Program, apply new data collected for each of the three studies, and integrate extant data (MAX) for context and comparison.

3. Uses of Information Collected

The 9-year cross-site evaluation of the GLS Suicide Prevention Program has resulted in the largest repository of youth suicide prevention data in the United States. Across its history, the evaluation has responded to the National Strategy for Suicide Prevention (National Strategy) developed by the National Action Alliance for Suicide Prevention (Action Alliance) in 2001 and revised in 2012. Further, the evaluation aligns with and provides data sources to track SAMHSA’s Strategic Initiatives. The information gathered has been essential to SAMHSA and others in helping communities and decision-makers at all levels of government improve suicide prevention effectiveness. Building on the revised National Strategy (HHS, 2012) the Action Alliance released the first-of-its-kind action plan in 2014, A Prioritized Research Agenda for Suicide Prevention: An Action Plan to Save Lives, aimed at prioritizing suicide prevention research with the greatest likelihood of reducing suicide morbidity and mortality. Consistent with the Action Alliance goal to save 20,000 lives in 5 years, the agenda outlines multiple approaches that collectively could achieve a reduction in suicide attempts and deaths by 20% in 5 years and 40% or greater in 10 years (Action Alliance, 2014). The agenda is organized around 6 key questions, each tied to one or more of 12 aspirational goals which serve as an organizing frame for suggested research pathways to reduce the burden of suicide. The public health approach to suicide prevention recommended in these guiding documents has been a hallmark of SAMHSA’s suicide prevention programs. Information collected through the evaluation of the GLS Suicide Prevention Program has contributed to key areas of both efforts and is a priority for the NOE.

1. SAMHSA’s Strategic Initiatives

In FY 2010, SAMHSA developed an updated strategic plan, Leading Change: A Plan for SAMHSA’s Roles and Actions: FY 2011–2014. The 2011–2014 plan comprises 8 Strategic Initiatives (SIs) that focus on SAMHSA’s work on improving lives and capitalizing on emerging opportunities. In the Leading Change Accomplishments Report, SAMHSA reported attaining signification success in all 8 areas and meeting several goals within the SIs. Moreover, SAMSHA accomplished key objectives in three areas—Public Awareness and Support; Military Families; and Data, Outcomes, and Quality. It is these within these areas that information collected through the cross-site evaluation has contributed explicitly to SAMHSA’s success. In FY 2014 and based on accomplishments to date, SAMHSA updated its strategic plan, entitled Leading Change 2.0: Advancing the Behavioral Health of the Nation 2015-2018 (Leading Change 2.0). Leading Change 2.0 identifies 6 SIs. In particular, the NOE is in line with S-1 and its proposed strategies. Information collected through the NOE will help SAMHSA to assess these essential strategies aimed at influencing comprehensive change across the behavioral health service system, as well as assist SAMHSA in its aim to implement more rigorous evaluations of process, outcomes, and impacts of its initiatives.

SI-1: Prevention of Substance Abuse and Mental Illness

• Employ a continuum of care approach (promotion, prevention, treatment, and recovery strategies) to address behavioral health problems; COC and SSE Studies

• Increase awareness about risk and protective factors; Training, COC, and SSE Studies

• Develop and implement evidence-based practices for early identification; Training, COC, and SSE Studies

• Improve cultural awareness and competency to address health disparities; Training and COC Studies

• Utilize community coalitions to foster community health; COC and SSE Studies

2. National Strategy for Suicide Prevention (National Strategy)

The 2010 report, Charting the Future of Suicide Prevention, was commissioned by SAMHSA to assess the progress in suicide prevention since the release of the 2001 National Strategy. In addition to major achievements, the report highlighted the increased training of gatekeepers and enhanced collaboration across the public and private sectors as important areas of progress. In 2012, the National Strategy was revised and expanded to include 4 strategic directions, 13 goals, and 60 objectives. The 2012 National Strategy represents the culmination of an intensive consultation process coordinated by the Action Alliance, a nationwide partnership created in 2010 and comprised of more than 200 representatives from the public and private sectors. The revised NSSP has been updated to “reflect advances in suicide prevention knowledge, research, and practice, as well as broader changes in society and health care delivery that have created new opportunities for suicide prevention” (HHS, 2012).

The 2012 National Strategy responds to the needs of and aims to advance the field of suicide prevention and the wellbeing of persons at large. The hallmark of the strategy is its emphasis on incorporating prevention into all aspects of daily life. It emphasizes integrating suicide care into multiple sectors and settings, including those outside of behavioral health care, and its four priority areas (i.e., strategic directions [SDs]) and goals particularly relevant to the NOE.

SD-1: Healthy and Empowered Individuals, Families, and Communities (Training & COC Studies)

• Goal 1. Integrate and coordinate suicide prevention activities across multiple sectors and settings.

SD-2: Clinical and Community Preventive Services (Training & COC Studies)

• Goal 5. Develop, implement, and monitor effective programs that promote wellness and prevent suicide

• Goal 7. Provide training to community and clinical service providers on the prevention of suicide and related behaviors.

SD-3: Treatment and Support Services (COC & SSE Studies)

• Goal 8. Promote suicide prevention as a core component of health care services.

• Goal 9. Promote and implement effective clinical and professional practices for assessing and treating those identified as being at risk for suicidal behaviors.

SD-4: Surveillance, Research, and Evaluation (Training, COC, & SSE Studies)

• Goal 12. Promote and support research on suicide prevention.

• Goal 13. Evaluate the impact and effectiveness of suicide prevention interventions and systems and synthesize and disseminate findings.

3. Advancing the Field

Information gathered through the NOE will continue to be useful to SAMHSA and its partners, other Federal agencies and administrators, GLS State/Tribal and Campus grantees, legislators, the National Strategy and the field of suicide prevention, individual youth and their families, and the communities in which they live. Further, the focus of the NOE on assessing the implementation, outcomes, and impacts of the GLS Suicide Prevention Program will contribute immensely to advancing the field of suicide prevention. For example, data collected will help to inform SAMHSA’s other suicide prevention initiatives, such as the NSPL. Thus, the NOE will allow SAMHSA to expand the evidence base for suicide prevention; address factors contributing to suicide deaths and attempts; and establish standards for developing, implementing, and evaluating suicide prevention programs.

Without this evaluation, Federal and local officials will not determine whether the suicide prevention programs implemented as part of the GLSMA have an impact on the prevention of suicide; their effectiveness on identification, referral, and provision of services to youth and students identified as at risk; and whether GLS grantee programs are meeting the goals of the GLSMA. SAMHSA also will use the data collected to provide objective measures of its progress toward meeting targets of key performance indicators put forward in its annual performance plans as required by law under the GPRA.

3. Use of Improved Information Technology

Every effort had been made to limit burden on individual respondents who participate in the NOE through the use of technology. Data collection instruments will be administered via the Web and telephone. Below are descriptions of the technology that will be used for data collection: (a) Web-based data collection and management system, (b) CATI, and (c) SMS.

1. Web-based Data Collection and Management System

The Web-based data collection and management system, the Suicide Prevention Data Center (SPDC), will continue to be used to facilitate data collection by program staff, program participants, key stakeholders, students, and campus faculty/staff. The SPDC will serve as a (1) data entry tool for program and evaluation staff to enter information or data elements and (2) data collection tool for administering Web surveys to respondents. All data obtained through direct entry by program and/or evaluation staff or through Web surveys will be stored in the central repository to reduce evaluation burden on grantees and to allow ease of access to data for program personnel and evaluation team members. All data collected will be stored in the central data repository, allowing for the analysis and summary of information within and across surveys. To maintain privacy, the secure SPDC offers six levels of password-protected access to site-specific and aggregate data as described in Exhibit 6.

Only users with administrative privileges (evaluation management, evaluation team, and grantee site administrators) will have the security to access to raw data. To protect from potential misuse of those data (inadvertent identification of respondents as a function of their unique demographic/ workforce characteristic profile), the following measures are in place: (1) access to raw datasets will be restricted to designated individual(s) and (2) the grantee site administrator will sign a data use agreement. Within the context of protecting from inadvertent identification, this agreement will stipulate who, how, and under what circumstances the raw data can be analyzed/reported. For example, the NOE team will obtain an agreement from each grantee site administrator not to report categories in which less than 10 cases exist and to stipulate who will have access to raw data. Further, the agreement will indicate that no attempt, through complex analysis using outside information, will be made to ascertain the identity of particular persons from the datasets. A copy of the agreement is located in Attachment P.

Exhibit 6. SPDC User Security Levels

Security Level	SPDC Privileges
Evaluation Management Administrator	Capability to view number of instruments completed & submitted (all grantees) and to download site and aggregate datasets Access site-specific data from all grantees and available aggregate reports Capability to add, modify, and remove users for all grantees
Evaluation Team Member Administrator	Capability to view number of instruments completed & submitted (all grantees) Capability to download site and aggregate datasets Access site-specific data from all grantees and available aggregate reports
Grantee Site Administrator	Capability to view number of instruments completed & submitted and to download site-specific datasets Capability to enter data via upload or Web survey Access grantee-specific data and reports and available aggregate reports Capability to add, modify, and remove users for site
Grantee Site User	No access to view the number of instruments completed & submitted No access to datasets Access to aggregate reports only Capability to enter data via upload or web-survey
Grantee Contact User	Capability to enter data into the Web-based system only No other privileges (for data collectors and survey respondents)
SAMHSA, its Consultants, & Partners	Capability to view number of instruments completed & submitted (all grantees) Access site-specific data from all grantees and available aggregate reports

A System of Records Notice (SORN) and an HHS Privacy Impact Assessment (PIA) (see Attachment Q) form have been completed for the personally identifiable information that will be contained within the SPDC (related to SPDC users and the SMSS). The SORN was submitted in March 2013, and the PIA was submitted in November 2013.

2. Computer Assisted Telephone Interviewing (CATI) Technology

All TUP-S versions are telephone surveys that will be administered using CATI technology. The evaluation division of ICF, Public Health and Survey Research (PHaSR), operates fully integrated call centers in Burlington, Vermont; Seattle, Washington; and Martinsville, Virginia that together offer 340 CATI stations that conduct more than 600,000 interviews annually. The centers are networked with each other, use the same CATI software, operate on the same platform, and are connected by a high-speed link that allows projects managed in one site to be managed from another, or from both sites simultaneously. The CfMC questionnaire programming language provides call management and quota controls, inbound calling capabilities, multilingual interviewing capabilities, data back-up and monitoring, and incidence tracking. All CATI stations are equipped with predictive dialing capabilities. The use of the PHaSR CATI system, predictive dialing system, and supervisory staff ensure that this data collection is methodologically consistent with other study efforts.

3. Short Message Service (SMS) Technology

The SMSS will be administered via SMS, or text messaging, technology. Text messaging is popular among younger Americans and is increasingly being adapted for research purposes, though the field is in its infancy. The medium has some limitations—questionnaires must be shorter (to limit break-offs or incomplete surveys), and questions must be shortened and/or reworded to fit within the SMS frame work (e.g., character limitations, brief instructions).

3. Efforts to Identify Duplication

The NOE team, in developing the data collection activities and updated design for the evaluation, conducted a literature review to avoid duplication in data collection activities and the use of similar information. Specifically, existing research studies and the efforts of other Federal initiatives designed to evaluate suicide or suicide prevention were reviewed.

1. Existing Research

Many in the field of suicide prevention agree that there is insufficient information on the causes of suicide and even less information on how to most effectively prevent suicide (SPAN USA, Inc., 2001; Institutes of Medicine, 2002; U.S. Public Health Service, 2001). The studies on suicide prevention activities have provided important information, but for the most part have been conducted with nonrandomized groups. Similarly, the lack of longitudinal and prospective studies has been a barrier to understanding and preventing suicide (Institutes of Medicine [IOM], 2002). Acknowledging the dearth of information on the effectiveness of suicide prevention programs, the IOM Report, Reducing Suicide: A National Imperative, provided recommendations for increasing research on suicide (2002), including Federal funding for suicide prevention interventions and longitudinal studies that focus on the medium to long-term impacts of suicide prevention activities.

Since then, research has identified gatekeeper training as a critical element in suicide prevention efforts (Isaac et al., 2009) and also has shown that training increases knowledge, skills, an intention to help someone at risk for suicide among an array of gatekeepers (King & Smith, 2000; Reis & Cornell, 2008; Wyman, 2008; Keller et al., 2009; Matthieu, Cross, Batres, Flora, & Knox, 2008; Isaac et al., 2009. Initial findings from the cross-site evaluation of the GLS program indicate a positive collective impact of suicide prevention trainings on subsequent identification behavior of trainees (Condron, Godoy-Garraza, Walrath, McKeon, & Heilbron, 2014) and establishes the effect of GLS Suicide Prevention Program trainings on youth suicide attempts and suicide mortality. Findings indicate that counties where GLS trainings were implemented had lower suicide rates in the year following training events compared to similar counties that did not have GLS trainings (Walrath, Godoy-Garraza, Reid, Goldston, & McKeon, in press; SAMHSA, 2013b). Findings such as these helped to guide the direction of further research, such as determining the training types and practices that are most effective when identifying and referring youth.

In addition, the evaluation of SAMHSA’s National Suicide Prevention Lifeline (NSPL) has developed evidence to support crisis lines’ effectiveness and the value of suicide prevention efforts.² As a result, the NSPL has emerged as a key component for a range of suicide prevention programs and has expanded to offer clinical follow-up services to callers and to those who had received service in emergency departments and hospitals for suicidality.

2. Other Federal Efforts

The Centers for Disease Control and Prevention’s (CDC) National Center for Injury Prevention and Control provides funding and TA to states through its Core Violence and Injury Prevention Program (Core VIPP). The program supports State health departments in strengthening their capacity to collect data and use data for a better understanding of local injury issues, including suicide. The focus of Core VIPP is on supporting funded state partners in their efforts to build a solid violence and injury prevention infrastructure, collect and analyze data, and implement and evaluate injury prevention programs. This CDC program may provide a broader understanding of suicide as a by-product of its efforts to gain a better understanding of local injury issues, however, the focus of the GLS cross-site evaluation is specifically to evaluate the effectiveness of suicide prevention programs.

SAMHSA is sponsoring an ongoing evaluation of the NSPL. The purpose of the evaluation is to assess the impact of the national crisis hotline connecting callers to mental health professionals and to assess participation with the Lifelines networks. The specific focus of the ongoing evaluations changes as the components and dimensions of the National Suicide Prevention Lifeline expand. Although the data collection activities planned as part of this effort will provide valuable information on the effectiveness of this important service for at-risk youth, the scope of the evaluation focuses on all callers (adult and youth) to the national hotline and is specific to certain interventions. The NSPL has emerged as a key component for a range of suicide prevention programs and has expanded to offer clinical follow-up services to callers and to those who had received service in emergency departments and hospitals for suicidality. Based on this work and to continue to address gaps in research and the field, the NSPL will focus on evaluating clinical follow-up for those who received emergency department or hospital care, emerging technology approaches (i.e., chat interventions), and imminent risk caller guidelines.

In 2014, SAMHSA awarded cooperative agreements to twenty federally recognized tribes or tribal organizations to implement the Tribal Behavioral Health Program (Native Connections) to address high rates of suicide and substance abuse with AI/AN youth up to age 24. The NC program provides support to tribes and tribal organizations to build capacity for the implementation of suicide prevention, substance abuse prevention, surveillance, and mental health promotion activities among young people. For this new project, SAMHSA is conducting evaluability assessments to assess NC grantee readiness to participate in local or cross-program evaluation, as well as to support NC grantees in their efforts to implement and use existing and new surveillance to understand prevalence and prevention outcomes related to suicide and substance abuse.

Also in 2014, SAMHSA funded 4 National Strategy Grants to State program grantees to support the implementation of the 2012 National Strategy. The cooperative agreements provide funding for suicide prevention among working-age adults from 25 to 64 years old.

3. Impact on Small Businesses or Other Small Entities

Some data collection activities involve individuals from public agencies, such as mental health, juvenile justice, education, and child welfare, as well as colleges and universities. While most data will be collected from public agencies or universities, it is possible that organizations involved in the referral networks would qualify as small entities. While most likely employed by public agencies, respondents to the TUP-S, TUP-S RCT, RNS, and BHPS may also be employed by small businesses or other small entities, but these data collections will not have a significant impact on the agencies or organizations.

3. Consequences if Information Collected Less Frequently

The rigor of the NOE design and its ability to answer the primary evaluation questions are dependent on the frequency of the data collected. Additionally, because the NOE is aligned with the foci of the GLS State/Tribal and Campus Programs, the frequency with which data collection activities are administered is critical to SAMHSA’s overall assessment of the GLS Suicide Prevention Program. Exhibit 7 describes the consequences if data are collected less frequently.

Exhibit 7. Data Collection Activities and Consequences If Information Collected Less Frequently

Activity	Rationale
PSI	Grantees will be required to complete the PSI beginning in year 1 of the grant. Thereafter, they will complete the PSI on a quarterly basis over the duration of their grant period. Collecting this information quarterly is necessary to track progress toward meeting suicide prevention goals and to provide information on the development stage of products and services within State/Tribal and Campus programs. The consequence of collecting the PSI less frequently is losing information related to all studies of the NOE—Training, COC, and SSE—as well as losing the ability to track progress over time.
TASP	Because gatekeeper training is a widely implemented suicide prevention strategy among State/Tribal and Campus grantees, aggregate basic information about trainings and trainee types and roles is necessary for SAMHSA to understand how grant funds are being utilized in support of training.
TUP-S	Information from the TUP-S informs the Training Study and tracks the effectiveness of trainings on participant knowledge and use of skills. The consequence of not collecting these data at the conclusion of the training experience include a loss of knowledge about the types of trainings and practices implemented, as well as the impact of training on the identification and referral of at-risk youth.
TUP-S RCT	The longitudinal implementation of the TUP-S RCT is the basis for the Training Study RCT. Collecting this information less frequently will impair the rigor of the experimental design and the ability to identify the trainings and training enhancements that are most effective.
EIRF-I	The EIRF-I requires State/Tribal grantees to share existing data on the youth identified as at risk. Data from the EIRF-I are integral to understanding the impact of gatekeeper training on identifications and referrals and services received as a result of the GLS Suicide Prevention Program.
EIRF-S	State/Tribal grantees are also required to report aggregate screening information for all youth screened as part of their suicide prevention programs. The information collected includes the number youth screened, the number screening positive, and the number confirmed to be at risk. This information is necessary for SAMHSA to understand the types and effectiveness of screenings implemented as a result of the GLS program.
RNS	In order to complete the previously-approved RNS data collection protocol, two administrations of the RNS are important to assessing how networks influence referral mechanisms and service ability, as well as policies and protocols regarding follow-up for youth who have attempted suicide and are at risk for suicide. The consequences associated with less frequent data collection would be a lack of information that assesses the impact of time on the development of referral networks and coalitions. Data also examines how collaboration and integration are used to share and transfer knowledge, resources, and technology among State/Tribal grantees and stakeholders.
BHPS	The BHPS will inform SAMHSA about referrals for at-risk youth, the implementation of SSE care practices by behavioral health providers partnering with GLS grantees, and client outcomes. Information collected through the BHPS is critical to understanding the care practices implemented by partner providers—the primary providers of behavioral health care to youth and students identified as at risk through the GLS program. Data are collected once per year.
SMSS	In order to complete the previously approved SMSS data collection protocol, two administrations (grant years 1 and 3) of the cross-sectional SMSS are necessary to determine any change in awareness and knowledge as a result of suicide prevention activities. If collected less often, there would be no means by which to assess change over time, an important element of the suicide prevention program.

3. Consistency with the Guidelines of 5 CFR 1320.5(d)(2)

The data collection fully complies with the requirements of 5 CFR 1320.5(d) (2).

3. Consultation Outside the Agency

1. Federal Register Notice

SAMHSA published a notice in the Federal Register on October 7, 2015 (80 FR 60694), soliciting public comment on this study. No public comments were received.

2. Consultation Outside the Agency

Consultation on the design, instrumentation, and statistical aspects of the evaluation has occurred with individuals outside of SAMHSA. In 2005, an evaluation steering committee was established to provide input and guidance in designing and implementing the original cross-site evaluation. Consultation with the steering committee has continued since 2005. Similarly, an evaluation advisory Expert Advisory Panel (EAP) established in 2014 and convened in 2015 to provide input and guidance on the updated design and implementation of the NOE. Consultation with this EAP will continue as needed throughout the grant-funding period. Representatives on the EAP include leaders in the field of suicide prevention program implementation, research, and evaluation. In addition, a panel of local evaluators from currently funded grantees that will be participating in the NOE reviewed the evaluation and provided feedback in 2015. The SPRC provides TA to entities implementing suicide prevention programs. Input from representatives of the CDC was also solicited in 2005. The CDC has conducted research in the field of suicide prevention and was consulted to comment on the cross-site evaluation design, frequency of data collection activities, and instrumentation.

As with previous evaluations, updates to the instruments were informed through direct consultation with current and former grantees, as well as representatives of the SPRC and CDC. These consultations had four purposes: (1) to ensure continued coordination of related activities, especially at the Federal level; (2) to ensure the rigor of the evaluation design, the proper implementation of the design, and the technical soundness of study results; (3) to verify the relevance and accessibility of the data to be collected; and (4) to minimize respondent burden.

3. Payment to Respondents

As with previous evaluations, the NOE will use a research-based approach and will require participation students, suicide prevention training participants, and others beyond grantee program staff. Consequently, remuneration is suggested for respondents not directly affiliated with suicide prevention programs at the time of their participation in surveys and interviews as compensation for the additional burden, potential inconvenience of participation, and any related costs (e.g., transportation costs, mobile phone minutes or data, compensation for time). Remuneration also is a standard practice in longitudinal studies partly because respondents are typically not directly affiliated with the program being evaluated. Given the use of longitudinal data collection for the NOE and the hard-to-reach nature of these populations, compensation will be provided for activities that involve follow-up interviews. Historically, remuneration has been used with the TUP-S ($10) and the SMSS ($5). Moving forward, remuneration will continue for these activities and also is proposed for the TUP-S RCT ($10) and BHPS ($10) at the time of participation. Respondents to other data collection activities are primarily staff of the suicide prevention programs or close affiliates. Therefore, no remuneration is planned for those activities.

3. Assurances of Confidentiality

To ensure the confidentiality of data compiled the protection of human subjects, the data collection protocol and instruments for NOE will be reviewed through the ICF institutional review board (IRB) prior to the collection of covered or protected data. The ICF IRB holds a Federal wide Assurance (FWA00000845; Expiration, April 13, 2019) from the HHS Office for Human Research Protections (OHRP). This review ensures compliance with the spirit and letter of HHS regulations governing such projects. All protected data will be stored on secure servers at ICF in the manner described in the IT Plan and IT Data Security Plan submitted to and approved by SAMHSA January 7, 2015. In addition, the Web-based data collection and management system, the SPDC, will facilitate data entry and management for the evaluation.

Descriptive information will be collected from respondents to data collection activities, but no identifying information will be entered or stored in the SPDC. All hard copy forms with identifying information will be stored in locked cabinets; the contact information will be entered into a password-protected database that can only be accessed by the limited number of individuals who require access (selected ICF staff such as telephone interviewers, data analysts and administrative staff for administering the incentives). These individuals have signed privacy, data access, and data use agreements. Identifying information collected to facilitate the administration of surveys will not be stored with survey responses. Further, datasets will be stripped of any identifying information prior to use by data analysts. Once incentives are sent out (as appropriate), respondent contact information will be deleted from the database and the hard copy forms will be destroyed.

Data collection activities requiring the collection of identifying information for the NOE include the following: PSI, TUP-S, TUP-S RCT, RNS, BHPS, and SMSS. Specific procedures to protect the privacy of respondents are described below.

• PSI: Information to complete the inventories will be directly entered into the Web-based system. To access the system, respondents receive an individual username and password to protect their privacy and no identifying information is requested on the inventories.

• TUP-S: Prior to each training event, all training participants will be asked to respond to a request for consent to contact for the TUP-S that will gather information about identification and referral behaviors to help establish a baseline about trainee behaviors. The consent-to-contact form will include a training participant ID (which contains no identifying information) and ask participants to provide the identifying information (name, telephone number, and mailing address) necessary to contact them for the survey and administer the incentive. A link to an online form will be shared with grantees to gather this information at registration prior to the trainings. Alternatively, grantees and training facilitators can collect the information before the training activity via hard-copy form or by sharing the link to the online form (which will be mobile device compatible). Information gathered via hard-copy form will be returned to ICF (or submitted online to ICF). Trainees who are assessed at 3 months will be asked for their consent to be contacted for a follow-up at 6 months. Verbal consent will be obtained at the start at the start of survey administration.

• TUP-S RCT: Grantees participating in the Training Study RCT will share the TUP-S RCT consent-to-contact form with prospective participants at registration. ICF will e-mail a link to an online consent-to-contact form to facilitate the process for those sites that offer online registration; this will send information electronically to ICF. If hard-copy consent-to-contact forms are distributed, the grantee will gather the forms and send them to ICF for processing. Trainees who participate at baselines will be asked for their consent to be contacted prior to the 3-, 6-, and 12-month surveys. Verbal consent will be obtained at the start of survey administration.

• RNS and BHPS: Identifying information for respondents to the RNS and BHPS will be necessary for survey administration. Contact information will be limited to agency affiliations, names, email addresses and telephone numbers and will be entered into a password-protected database. Although the individual’s identifying name will not be used by any reports or datasets, the reports and datasets will contain the name of the agency/organization and the information provided about the agency or organization. Therefore, an individual may be identifiable when reporting results. Respondents are informed of possible identification in the consent language at the start of the Web-surveys. To capture the potentially lengthy set of NPIs associated within provider organizations (to link to the Medicaid claims data for analyses), the BHPS will include a link so that providers can upload a file containing NPIs rather than entering them in the survey form.

• SMSS: Identifying information will be necessary to contact respondents via text message and will be limited to a mobile phone number. Campus grantee staff will assist in recruiting students for the baseline SMSS. Each student whose number has been obtained will receive an initial text message asking if he or she would like to participate in a text survey about suicide prevention. Upon responding “yes,” the student will receive 2-3 messages containing consent language, including background information about the study, risks and benefits, as well as information to contact the NSPL if they require help. After the consent process, students will have the option to select “yes” to continue to the survey. Identifying information will not be used for analysis or reporting efforts.

3. Questions of a Sensitive Nature

Survey and interview instruments include questions that are potentially sensitive because this project concerns suicide prevention. These questions collect information about mental health, substance abuse, family circumstances, mental health services seeking, and suicide. These questions are central to the agency’s goal of learning about the protective factors and campus wellness context related to suicide prevention. Names and email addresses collected as part of the consent process will be kept separate from responses as stated above. All data will be managed and stored in the manner described above and therefore will be unavailable to anyone but authorized project staff. Active consent forms explicitly advise potential respondents and participants about the sensitive nature and content of the data collection protocol as well as the voluntary nature of all data collection activities. Unanticipated or negative consequences will be reported immediately to the campus and ICF institutional review boards (IRB). The Principal Investigator and Project Director will also consult with appropriate clinical professionals and immediately determine if the participant presents a risk to themselves or others and make appropriate referrals. The proposed protocols were submitted to the ICF IRB for review and were approved on date. Updated protocols will be submitted to the IRB once OMB approval is received.

3. Estimates of Annualized Burden Hours and Costs

Clearance is being requested for 3 years of data collection for the National Outcomes Evaluation. Data collection for the currently active grantees (104 total) is operating under the previously approved clearance (OMB No. 0930-0286), which will expire January 31, 2017.

Exhibit 8 below describes the burden and costs associated with NOE data collection activities. Burden is calculated for 49 State/Tribal and 91 Campus grantees, which represents the number currently active (104) and the number expected to be funded in FY 2016 (21) and FY 2017 (15). All Campus grantees, as well as the 7 State/Tribal grantees funded in FY 2013, have 3-year funding cycles. Beginning in FY 2014, SAMSHA expanded State/Tribal grants to a 5-year funding cycle. This is taken into account in the table below. The cost was calculated based on the hourly wage rates for appropriate wage rate categories using data collected as part of the National Compensation Survey (BLS, 2014) and from the U.S. Department of Labor Federal Minimum Wage Standards. Exhibit 9 shows an annualized summary of burden hours by respondent type.

Exhibit 8. Estimated Annualized Burden Hours and Costs (Across the 3-Year Clearance Period)

Type of Respondent	Instrument	Number of Respondents	Responses per Respondent	Total Number of Responses	Burden per Response (hours)	Annual Burden (hours)	Hourly Wage Rate ($)	Total Cost ($)
State/Tribal Instruments
Project Evaluator	PSI	43	4	172	0.750	129	36.72	4,737
Project Evaluator	TASP	43	4	172	0.250	43	36.72	1,579
Project Evaluator	EIRF-Individual Form	43	4	172	0.750	129	36.72	4,737
Project Evaluator	EIRF Screening Form	43	4	172	0.750	129	36.72	4,737
Provider Trainee	TUP-S Consent to Contact	6,000	1	6000	0.167	1000	22.22	22,220
Provider Trainee	TUP-S 3 Month Version	2,000	1	2000	0.500	1000	22.22	22,220
Provider Trainee	TUP-S 6 Month Version	600	1	600	0.417	250	22.22	5,555
Provider Trainee	TUP-S RCT BL Version	444	1	444	0.417	185	22.22	4,111
Provider Trainee	TUP-S RCT 3 Month Version	444	1	444	0.500	222	22.22	4,933
Provider Trainee	TUP-S RCT 6 Month Version	444	1	444	0.417	185	22.22	4,111
Provider Trainee	TUP-S RCT 12 Month Version	444	1	444	0.417	185	22.22	4,111
Provider Stakeholder	RNS	26	1	26	0.667	17	36.44	620
Behavioral Health Provider	BHPS	407	1	407	0.750	305	36.44	11,114
Campus Instruments
Project Evaluator	PSI	56	4	224	0.750	168	36.72	6,169
Project Evaluator	TASP	56	4	224	0.250	56	36.72	2,056
Project Evaluator	SBHF	56	1	56	0.667	37	36.72	1,359
Student	TUP-S Campus Version	167	1	167	0.167	28	7.25	203
Student	SMSS	734	1	734	0.083	61	7.25	442
Total		12,050		12,902		4,129		$105,014

Exhibit 9. Annualized Summary Burden by Respondent Type

Respondents	Number of Respondents	Responses/ Respondent	Total Responses	Total Annualized Hour Burden
State/Tribal Instruments
Project Evaluators	43	16	688	430
Provider (Trainees)	10,376	1	10,376	3,027
Behavioral Health Provider/ Provider Stakeholder	433	1	433	322
Campus Instruments
Project Evaluators	56	9	504	261
Students	901	1	901	89
Total	11,809	-	12,902	4,129

3. Estimates of Annualized Cost Burden to Respondents or Record Keepers

Grantees are collecting the majority of the required data elements as part of their normal suicide prevention program operations. Grantees will maintain this information for their own program planning, quality improvement, and reporting purposes. Therefore, there are no additional capital or start-up costs associated with the NOE. There will be some additional burden on record keepers to provide potential respondent lists for data collection activities. However, these operation costs will be minimal. Other costs related to this effort, such as the cost of shipping consent-to-contact forms is cost to the Federal Government as part of the funding received for participation in the evaluation. Each grantee has been funded, as part of the overall cooperative agreement award, to fund an evaluator and related costs to carry out the requirements of the cross-site evaluation. Therefore, no cost burden is imposed on the grantee by this additional effort.

3. Estimates of Annualized Cost to the Government

CMHS has planned and allocated resources for the management, processing and use of the collected information in a manner that shall enhance its utility to agencies and the public. Including the Federal contribution to local grantee evaluation efforts, the contract with the National Evaluator, and Government staff to oversee the evaluation, the annualized cost to the Government is estimated at $4,339,803. These costs are described below.

Each grantee is expected to fund an evaluator to conduct the self-evaluation and to satisfy the requirements of the cross-site evaluation. It is estimated that participating in the NOE will require 0.20 full-time equivalent (FTE) to collect information, enter information into the Web-based data collection and management system, and to conduct analyses at the local level. Assuming: (1) an average annual salary of $76,378 (BLS, 2014) for a 0.20 FTE evaluator; (2) 45 State/Tribal and 59 Campus grantees; and (3) that Campus grantees had to cost share on a 1:1 basis, the annual cost for the NOE at the grantee level is estimated at $1,680,316. These monies are included in the cooperative agreement awards.

The NOE contract has been awarded to ICF for evaluation of the GLS Suicide Prevention Program. The current evaluation contract with SAMHSA is funded to conduct the NOE with 104 grantees over the next 5 years with a value of $13,167,432. Additional contract dollars will be set aside for the evaluation as future cohorts are funded. The estimated average annual cost of the contract will be $2,633,487. This covers expenses related to developing and monitoring the NOE including, but not limited to: developing the evaluation design and instrumentation; developing training and TA resources (i.e., manuals, training materials, etc.); conducting in-person or telephone training and TA; monitoring of grantees; traveling to grantee sites and relevant meetings; and analyzing and disseminating data activities. In addition, these funds will support the development of the Web-based data collection and management system and fund staff support for data collection. It is estimated that CMHS will allocate 0.30 of a full-time equivalent each year for Government oversight of the evaluation. Assuming an annual salary of $80,000, these Government costs will be $24,000 per year.

3. Changes in Burden

Currently there are 8,014 annual burden hours in the OMB inventory. SAMHSA is requesting 4,129 annual burden hours for this submission, representing a decrease of 3,885 annual burden hours. The total number of grantees for which burden is calculated is 140 (49 State/Tribal grantees and 91 Campus grantees), which includes the number currently active (104) and those expected to be funded in FY 2016 (21) and FY 2017 (15). Other program changes that account for the change in burden include:

• The previously-approved TUP-S Adolescent Version will no longer be administered with State/Tribal grantees. The estimated annual burden for this effort was 48 hours.

• The previously-approved Coalition Profile will no longer be administered with State/Tribal grantees. The estimated annual burden for this effort was 11 hours.

• The previously-approved Coalition Survey will no longer be administered with State/Tribal grantees. The estimated annual burden for this effort was 286 hours.

• The previously-approved TES Individual Forms will no longer be administered with Campus grantees. The estimated annual burden for this effort was 737 hours.

• The previously-approved LAFI will no longer be administered with Campus grantees. The estimated annual burden for this effort was 18 hours.

• The previously-approved SAIS will no longer be administered with Campus grantees. The estimated annual burden for this effort was 3,200 hours.

• The previously-approved TUP-S Campus will be conducted in OMB Year 1 with Campus Cohorts 7 and 8 only and discontinued after. The estimated annual burden for the TUP-S Campus has decreased from 59 hours to 28 hours.

• The previously-approved RNS will be administered in OMB Year 1 with Cohort 8 State/Tribal grantees only and discontinued after. The estimated annual burden for the RNS has decreased from 956 hours to 17 hours.

• The previously-approved SMSS will be conducted in OMB Year 1 with Campus Cohort 7 grantees only and discontinued after. The estimated annual burden for the SMSS has decreased from 432 hours to 61 hours.

• The addition of the TUP-S RCT and BHPS represent an annual increase of 777 hours and 305 hours, respectively, for a total increase of 1,082 annual burden hours.

3. Time Schedule, Publication, and Analysis Plans

1. Time Schedule

The time schedule for implementing the cross-site evaluation is summarized in Exhibit 10. A 3-year clearance is requested for this project.

Exhibit 10. Time Schedule

Activity	Timeframe
Begin data collection for 104 grantees 45 S/T and 59 Campus grantees	February 2016 (1 month after OMB clearance estimated for January 2016)
Data collection completed for FY2013 funded grantees S/T Cohort 8 and Campus Cohort 7 grantees close out Data collection requirements for the RNS, TUP-S Campus, and SMSS fulfilled	September 2016
New funding expected for 21 FY2016 grantees Data collection begins for new Cohort 10 S/T grantees (4) and Cohort 9 Campus grantees (17)	October 2016
Data collection completed for FY2014 funded grantees S/T Cohort 9 and Campus Cohort 8 grantees close out	September 2017
New funding expected for 15 FY 2017 Campus grantees Data collection begins for new Cohort 11 Campus grantees (15)	October 2017
Data collection completed for FY2015 funded grantees Campus Cohort 9 grantees close out	September 2018

2. Publication Plans

The GLSMA requires annual congressional reports summarizing the results of NOE. The evaluation team will analyze data collected and prepare annual congressional reports to summarize key findings. A final report on the results of the NOE is also required by the GLSMA, and will be produced by the evaluation team. Because of the importance of the NOE to the field of suicide prevention, in collaboration with SAMHSA and the Government project officer, the results of the evaluation also will be published in relevant professional journals to inform the research community as well as the decision making of policymakers and program administrators.

ICF will develop a minimum of one GLS-focused article for submission to peer-reviewed journals for each year of the contract. An outline and or draft for each manuscript will be submitted to the COR for review and feedback on the structure and content and the potential peer-reviewed journals for submission. Potential manuscript topics that will be explored across the contract will include findings related to priority areas, such unintended consequences of GLS Suicide Prevention Programs. In FY 2015, the publications will likely focus on findings related to understanding the reduction in suicide mortality in youth following implementation of school-based suicide prevention strategies and examining the anticipated impact of using longer gatekeeper trainings versus brief gatekeeper trainings. In addition, or for early Year 2, an article may be developed from the presentation on GLS Campus Program prevention and suicide risk behavior among students.

Potential manuscript topics that will be explored across the contract will include findings related to priority areas, such as unintended consequences of GLS Suicide Prevention Programs. Manuscript topics may also be related to research questions and findings of the GLS National Outcomes Evaluation on training, continuity of care, and suicide safer environments. This may include the following topics:

• Further examination of the impact of training and prevention strategies on potential reduction in suicide mortality and attempts in youth

• Understanding the anticipated impact of incorporating behavioral rehearsal in every brief gatekeeper training and/or booster intervention on trainee identification behaviors

• Understanding the anticipated impact of implementing Zero Suicide in mental health organizations

• Further exploration of potential unintended consequences of youth suicide prevention (e.g., number of adults served, dissemination of prevention information among trainee personal/professional networks)

All publications will be submitted to the Contracting Office Representative (COR) in draft form for review and approval prior to submission to the selected journal. Examples of journals that will be considered as vehicles for publication include the following:

• American Journal of Public Health

• American Psychologist

• American Journal of Diseases of Children

• Child Development

• Crisis

• Evaluation Review

• Evaluation Quarterly

• Journal of the American Academy of Child and Adolescent Psychology

• Journal of Applied Development Psychology

• Journal of Child and Family Studies

• Journal of Clinical Child and Adolescent Psychology

• Journal of Consulting and Clinical Psychology

• Journal of Health and Social Behavior

• Journal of Mental Health Administration

• Psychological Reports

• Social Services Review

• Suicide and Life Threatening Behavior

3. Data Analysis Plan

Data collected through the three NOE studies will be analyzed to address key evaluation questions and related subquestions (see Section A.2.a). Analysis plans for each study are described below. In addition, three special analyses will be conducted to address evaluation questions that cut across the NOE studies. These analyses integrate existing data sources, findings from the core and enhanced studies, and data collected since 2005 from earlier GLS cohorts (included in previous evaluations of the GLS portfolios). Each special analysis is associated with a primary evaluation question and multiple subquestions.

• The ecological analysis of impact compares mortality and attempts in areas exposed and areas not exposed to GLS program activities.

• Outcome modeling will be used to anticipate the impact of particular interventions taking advantage of existing evidence and evidence gathered through the NOE.

• The unintended consequences analysis will explore and when possible, quantify the magnitude of consequences of program implementation beyond those directly related to suicide prevention.

1. Training Study Analysis

EQ-1: Are certain training approaches effective in building capacity to increase youth identification (when compared with more basic trainings)?

Training Study Core

Descriptive statistics will be used to provide a precise characterization of training activities, their outputs, and proximal outcomes. This includes the (1) number of trainings implemented by training type; (2) number of trainees participating in the activities, their typical role, and the setting in which participants typically interact with youth; and (3) participants’ behavior after training, including number of identification referrals of youth at risk and awareness of follow-up to services received. Both trainees’ characteristics and posttraining behavior are collected from a probabilistic samples of participants. In such a case, appropriate measures of uncertainty (i.e., standard error and confidence intervals) will be computed and reported with their respective summary statistics.

Multivariate regression techniques, particularly ordinal and binary logistic regression, will be implemented to explore and summarize association between training activities and their proximal outcomes, as well as variation in outcomes by subpopulation of interest. For example, ICF has recently shown that, for participants typically interacting with youth in school settings, a larger number of identifications were associated with participation in longer gatekeeper training when compared with shorter trainings 3 months after the activity (Condron et al., 2014). The analysis controls for several participant characteristics, such as typical time interacting with youth. In that analysis, ICF did not find that the results were heterogeneous by training event or grantee. However, both factors could typically be sources of clustering (participants in the same event, or in trainings by the same grantee, could tend to have more similar posttraining behavior). These factors will continue to be tested and, if necessary, addressed through either mixed-effects models or the use of generalized estimating equation.

Training Study RCT

The Training Study RCT will rely on a factorial randomized design to assess, in real-life settings, the impact of two factors in enhancing the effectiveness of gatekeeper training: the incorporation of active learning techniques (in particular, role-playing exercise), and the use of a booster session 3 to 6 months following participation the initial training. Due to randomization, a comparison of proportions between arms (e.g., the proportion of participants who identify at least one youth at risk, refer the youth to support, and are aware whether the youth received the support) may be directly informative of difference in effectiveness. Additionally, regression analysis will be performed to identify variations in effectiveness by trainee baseline characteristics, such as trainee’s role, setting in which they typically interact with youth, and typical interaction time with youth. Difference in nonresponse patterns between arms is a common challenge encountered in a randomized study setting. Particularly challenging are situations where nonresponse is associated with the value of the unobserved outcome. In the present application, however, we do not anticipate the type of training assigned to predict response rates. Several methods are available to handle missing data if the data are missing completely at random or missing at random after baseline characteristics or previous values of the outcome are controlled for. Compliance, on the other hand, may plausibly be predicted by assignment (e.g., in the case of the booster intervention). Although, the so-called “intention-to-treat” analysis will still be informative, instrumental variable estimation will be implemented to estimate effectiveness associated with actual participation (as opposed to assignment to a type of training).

2. COC Study Analysis

EQ-2: Are GLS prevention activities effective in developing continuity of care from identification, to referral of at-risk youth, to the provision of needed services? Are early identification and referral practices associated with service receipt and follow-up treatment adherence compared with those not retained in services?

COC Study Core

ICF will rely on descriptive statistics to regularly provide a precise characterization of the program early identification, referral, and follow-up activities as well as youth proximal outcomes, including: (1) the number of youth identified as at risk for suicide by GLS grantees by youth characteristics (gender, race-ethnicity) and setting of identification; (2) the proportion of those youth identified at risk who received follow-up support; and (3) the risk status of the youth receiving mental health services by a GLS provider months after initial contact.

ICF will use multivariate regression techniques, particularly binary logistic regression, to explore and summarize association between proximal outcomes and youth, provider, and grantee characteristics. For example, ICF has shown that for youth identified by trained gatekeepers, age was associated with the probability of being referred to mental health services and, in turn, referrals to the emergency room were associated with a higher likelihood of receiving the service (Rodi et al., 2012). This analysis controlled for several additional gatekeeper and youth characteristics, including gatekeeper role and setting of identification. The analysis will incorporate the grantee as a possible source of variation in follow-up patterns, either through random-effects models, as in this example, or through the use of generalized estimation equation in combination with grantee-level fixed effects.

3. SSE Study Analysis Plan

EQ-3: Does the provision of services by GLS behavioral health provider networks implementing a zero suicide framework reduce suicide attempts, hospitalization, and associated costs (return on investment) compared with non-GLS behavioral health providers?

SSE Study Core

Across the various core instruments within the SSE, we will use descriptive statistics to provide a precise characterization of SSE activity implementation and associated outcomes. For campus grantees, this will include annual descriptive analyses of SBHF data to investigate changes in screening, assessment, follow-up, and services for students at risk of suicide, along with descriptive analyses of change in the number of suicide attempts and deaths over time. We will use multivariate regression techniques, particularly ordinal and binary logistic regression, to explore and summarize association between campus implementation of SSE practices, as measured by the SBHF and PSI, and the outcomes of suicide attempts and suicide deaths (from SBHF).

For State/Tribal grantees, we will conduct similar descriptive analyses of SSE care activities and associated outcomes. The BHPS provides detailed information on implementation of these care practices within the behavioral health providers partnering with GLS grantees. Variation over time in the implementation of elements of the suicide safer environment framework, as well as their association with provider characteristics, will be explored and summarized using multivariate regression techniques, particularly binary logistic regression. Qualitative analysis aims at classifying groups of providers by their approach to the implementation of suicide safer environments, and will use model-based clustering techniques (e.g., latent class analysis).

SSE Cohort Study (MAX Claims Analysis)

The SSE enhanced component uses a claims-based retrospective cohort design to compare youth outcomes, particularly nonfatal suicide attempts requiring medical attention, the year following a contact with a GLS behavioral health provider with the outcomes of youth who, under similar circumstances, enter into contact with non-GLS providers. Naturally, youth identified as at risk of suicide are not randomly assigned to services received from either a GLS or non-GLS provider. Instead, propensity score matching techniques will be used to identify control cases that share key characteristics with these GLS cases, including demographics, suicide-related diagnoses, previous suicide attempts, mental health service, and prescription use. In the case of hospital claims data, it is possible to determine whether the patient died during hospitalization. Using this subset of claims data, it will also be possible to conduct similar analyses comparing the rate of this subset of suicide deaths among youth in contact with GLS providers to youth in contact with non-GLS providers.

4. Ecological Analysis of Impact

EQ-4: Is there a reduction in suicide behavior in the areas exposed to GLS interventions compared with similar areas that were not exposed to those interventions?

The ecological analysis of impact relies on comparing the change in outcomes (mortality, suicide attempts) in areas exposed to GLS interventions with the change observed in areas that were not exposed to those interventions during the same period. Naturally, GLS grantees did not randomly decide the areas and timing for implementation of GLS interventions. Although different approaches exist to handle potentially confounding variables, propensity score–based techniques, including matching and subclassification, are appealing due to their reduced dependence on statistical assumptions. In addition, applying the same analysis to related variables that are not expected to be affected by GLS interventions (termed ‘control’ outcomes) offers additional protection against unmeasured confounding variables. Exhibit 11 outlines questions/subquestions and contract year for the Ecological Impact Analysis.

Exhibit 11. Ecological Impact Analysis Questions and Timeline

EQ-4: Is there a reduction in suicide behavior in the areas exposed to GLS interventions compared with similar areas that were not exposed to those interventions?	FY15	FY16	FY17	FY18	FY19
4.1: Is there a reduction in suicide mortality in youth aged 10–18 following implementation of school-based suicide prevention strategies during 2006–2011? Is the difference associated with the level of ‘saturation’ of teachers and school staff with trainings?	X
4.2: Is there a reduction in suicide mortality and attempts in youth aged 19–24 following the implementation of higher education–based prevention strategies during 2006–2011?		X
Other Potential Questions 4.3: Is there a reduction in suicide mortality and attempts in youth aged 19–24 following the implementation of GLS trainings in community settings? 4.4: Is there a reduction in self-inflicted injury requiring hospitalization in youth aged 10–24 following GLS training implementation in 20 States? (HCUP—State Emergency Department Databases/State Inpatient Databases; Truven)		X	X	X

The ecological analysis of impact takes advantage of the availability of three types of information for relatively small areas (e.g., county):

• Primary data from the NOE and previous cross-evaluation design measuring program activities and outputs since program inception

• Large extant data sets capturing suicide mortality and nonfatal attempts for long periods of time before GLS Suicide Prevention Program inception (i.e., Centers for Disease Control and Prevention’s [CDC’s] Compressed Mortality Files and SAMHSA’s National Survey on Drug Use and Health);

• Secondary data on a rich set of potential confounding variables, including unemployment, income, and health insurance coverage.

ICF has identified additional sources containing pertinent outcome information. Although promising, accessibility issues, particularly regarding key data elements that would allow the integration with primary data collected through the national evaluation (such as small area identifiers), have not been entirely resolved. Additional sources include: Healthcare Cost and Utilization Project (HCUP), Center for Collegiate Mental Health (CCMH), and the National College Health Assessment (NCHA). The ecological analysis of impact integrates information on program activities collected through the national evaluation with outcomes and covariate information from secondary sources. Sources and possible indicators by type of variable are presented in Exhibit 12.

Exhibit 12. Possible Indicators and Sources by Variable Type

Type of Variable	Indicator	Source
Suicidal behavior	Youth suicide rate	Compressed Mortality Files (CMF)
	Attempt rate among young adults	NSDUH
Local GLS implementation	Number of trainings and trainees in school, higher education, and community settings	TASP; TES
Covariates	Number of 5–12 grade teachers; higher education students enrollment	NCES-CCD
	Demographic characteristics	U.S. Census Bureau’s population estimates
	Proportion with no health insurance	U.S. Census Bureau’s Small Area Health Insurance Estimates (SAHIE)
	Poverty rate; median Income	Small Area Income and Poverty Estimates (SAIPE)
	Unemployment	Bureau of Labor Statistic, the Local Area Unemployment Statistics (LAUS)
	Six-level urban-rural classification scheme	CMF

The ecological analysis of impact relies on comparing the change in outcomes (mortality, suicide attempts) in areas exposed to GLS interventions with the change observed in areas that were not exposed to those interventions during the same period. Because of steps taken to increase comparability, ICF will be able to use a relatively simple regression model for the main analysis. The outcome measure (i.e., the suicide rate in each county and year) will be regressed on the independent variables (the measures of GLS implementation) using the weighted sample. Sensitivity of results will be assessed to extreme weights by refitting the regression after truncating 1% and 5% of the weights at each extreme of the distribution. Estimates will be created for all of the regression models using weighted generalized estimating equations with errors assumed to be clustered at the State level. Complementary approaches, such as models for the ‘demean’ outcome, will also be explored. These approaches may offer additional protection against the influence of unobserved, but time-fixed confounding variables.

5. Outcome Modeling Analysis

EQ-5: What is the anticipated impact (in terms of averted suicide attempts and suicide deaths) of specific GLS interventions (e.g., increased use of role-play in gatekeeper trainings, implementation of follow-up contact after identification at school)?

While direct evidence from RCTs or other strong study designs is generally thought to provide the strongest evidence of intervention effectiveness, statistical modeling can aid in understanding the impact of an intervention in a broad population (Lynch, 2014). In addition, modeling can help to systematically synthesize data from multiple sources and studies. Models clearly define alternatives, take outcomes under consideration, and make assumptions explicit so that they can be discussed and modified or tailored to specific needs or policy decisions. Since most studies of suicide prevention are small and relatively short term, modeling provides a way to begin to understand what the population impact of implementing effective interventions might look like over the long term. Exhibit 13 outlines the questions/subquestions and contract year for the Outcome Modeling Analysis.

Exhibit 13. Outcome Modeling Analysis Questions and Timeline

EQ-5: What is the anticipated impact (in terms of averted suicide attempts and suicide deaths) of specific GLS interventions (e.g., increased use of role-play in gatekeeper trainings, implementation of follow-up contact after identification at school)?	FY15	FY16	FY17	FY18	FY19
5.1: What is the anticipated impact of using longer gatekeeper trainings instead of brief gatekeeper trainings [Training Study]?	X	X
5.2: What is the anticipated impact of incorporating role-playing in every brief gatekeeper training and/or booster intervention following gatekeeper training [Training Study]?			X	X
5.3: What is the anticipated impact of incorporating follow-up contact after identification at school [Continuity of Care Study]?			X	X
5.4: What is the anticipated impact of implementing Zero Suicide in a number of mental health organizations [Suicide Safer Environment Study]?			X	X

The Action Alliance’s Research Agenda for Suicide Prevention included a set of models developed to determine the potential effects of alternative prevention interventions on suicide deaths (NAASP, 2014). A similar approach will be employed to take full advantage of the evidence gathered through the NOE by anticipating the impact of changes in the intervention, such as the widespread adoption of booster trainings, the implementation of a specific method to increase youth engagement with the recommended treatment, or the increased use of particular elements of the Zero Suicide framework by health providers. A preliminary list of interventions and outcomes (in terms of reduction of suicide attempts or suicide deaths), together with the study that will constitute the primary source, is presented in Exhibit 14. The primary data sources (i.e., TUP-S, SUESI, and BHPS) and associated methods of data collection were described in Section A.2.a.

Exhibit 14. Preliminary Interventions and Parameters by Primary Source

Intervention	Parameter	Source
Use longer gatekeeper trainings instead of brief gatekeeper trainings	Difference in gatekeeper behavior (identification, referrals, and follow) by type of training	TUP-S Training Core
Incorporate role-play in every brief gatekeeper training	Effect of the use of role-playing in gatekeeper behavior (identification, referrals, and follow-ups)	TUP-S Training RCT
Incorporate booster intervention following gatekeeper training	Effect of booster intervention on gatekeeper behavior (identification, referrals, and follow-ups)	TUP-S Training RCT
Behavioral health organizations implement Zero-Suicide interventions	Difference in attempt rates among patients served by behavioral health providers implementing Zero Suicide	BHPS; MAX SSE Cohort Study

Both literature review and the analysis of secondary data sources will be necessary to obtain population and event parameter estimates for each model to complement the findings from the GLS studies. Peer-reviewed journal articles, in particular, and published systematic reviews will be the main source for information on effectiveness of interventions beyond the sources collected in the national evaluation. For each type of intervention being modeled, a literature review will be conducted. Different secondary sources may be required to estimate population parameters depending on the final characteristics of the models. The list potentially includes all the sources previously identified for the Ecological Analysis of Impact, in particular the CMF and NSDUH. In addition, it is anticipated that Youth Risk Behavior Survey (YRBS) will be an important source of population parameters for school-aged youth. Youth report of behaviors include suicide ideation, planning, and attempt within the 12 months prior to the survey.

Outcome modeling will be used to anticipate the impact of particular interventions taking advantage of the evidence gathered through the national evaluation, as well as existing evidence. For the construction of each decision-analytic model we will use the following steps:

Step 1: Defining the specific question

• A preliminary set of specific questions has been identified covering key topics across the three evaluation studies. Further, an initial question has been selected that can take advantage of extant cross-site data (building on the analysis of Condron et al. [2014]) regarding comparative effectiveness of different types of gatekeeper trainings.

Step 2: Structuring a decision model

• This will rely on two of the most frequently used techniques for decision modelling (i.e., decision trees and Markov models). Decision trees will be developed primarily as an aid to structure the decision problem; however, a Markov cohort simulation model will be used for the final analysis. This model will build upon recently used models to study suicide prevention interventions (Lynch, 2014).

Step 3: Identifying and synthesizing evidence

• Priority will be given to take full advantage of information generated directly by the cross-site evaluation through the present design, as well as historically. However, both literature reviews, in particular published systematic reviews and the analysis of secondary data sources, will be necessary to obtained population and event parameter estimates (e.g., effectiveness of interventions).

Step 4: Run and test models under varying assumptions

• Reliable information on key population or event parameter may not be available. In those situations, different plausible assumptions may be posed. The sensitivity of the results to such estimates will be extensively examined.

6. Unintended Consequences Analysis

EQ-6: What are the positive and negative unintended consequences of suicide prevention activities (e.g., identification and referral of at-risk adults into care)?

In pursuing its main goal (i.e., reducing suicide among youth), the GLS Suicide Prevention Program will have consequences on population segments and public health issues beyond youth suicide. For example, the implementation of suicide safer health care environments will be beneficial for adults at risk of suicide as well as youth. An increased access to mental health services for youth in need will benefit youth with emotional, mental, or behavioral disorders, even if these youth would not have actually ended up committing suicide. An improvement in the coordination of the health care and support systems, which is necessary to increase continuity of care among youth at risk of suicide, will also be beneficial in addressing other public health issues. Although the evaluation data collection instruments were designed to address specific evaluation questions, they usually include measures that could support complementary analysis. ICF will systematically explore the information collected through the National Outcomes Evaluation across studies as well as secondary data to provide both qualitative and, when possible, quantitative assessments of the nature and extent of these consequences beyond the main GLS goal. Exhibit 15 outlines the questions/subquestions and contract year for the Unintended Consequences Analysis.

Exhibit 15. Unintended Consequences Analysis Questions and Timeline

EQ-6: What are the positive and negative unintended consequences of suicide prevention activities (e.g., identification and referral of at-risk adults into care)?	FY15	FY16	FY17	FY18	FY19
6.1: What are the potential unintended consequences documented in the literature? Is there any indication of magnitude of these consequences within GLS based on extant instruments?			X
6.2: What is the number of adults served by behavioral health providers partnering with GLS grantees? How frequently have GLS trainees used the training to work with adults at risk? How frequently have GLS trainees disseminated information from the training among their personal/professional network?				X

A preliminary list of potential unintended consequences by prevention strategy is presented in Exhibit 16. While the proposal is to expand and refine the list of potential benefits (costs) based on literature review, it is anticipated that only a limited number of these consequences can be explore.

Exhibit 16. Potential Unintended Consequence by Prevention Strategy

Strategy	Goal	Unexpected Benefits; Windfall Gain	Side Effects/ Iatrogenic Effects
Public awareness campaigns	Reduce stigma, increase help-seeking behavior associated with suicide	Reduce stigma, increase help-seeking behavior associated with mental health problems (beyond suicide)	‘Normalize’ suicide Increase unmet demand
Gatekeeper training	Increase identification (referral and treatment) of suicidal youth	Improve mental health among training participants Increase early identification (referral and treatment) of youth with emerging mental health or substance abuse problems Informal diffusion of knowledge to nonparticipants	Increase ‘false’ identifications Increase unmet demand Mismanagement of suicidal youth by laypersons Increase suicide ideation among participants
Screening	Increase identification (referral and treatment) of suicidal youth	Increase early identification (referral and treatment) of youth with emerging mental health or substance abuse problems	Increase ‘false’ identifications Increase unmet demand
Life-skills development activities	Increase suicide protective factors (e.g., copying skills)	Reduce risk of other mental health problems among youth
Suicide Safer Environment	Improve care of suicidal youth	Improve care of suicidal adults
Means restriction	Reduce suicidal youth access to lethal means	Reduce suicidal adult access to lethal means	Increase in suicide by alternative means or nonsuicide mortality (e.g., accidents)

Exhibit 17 presents possible sources and indicators related to some of unintended consequences presented above. The scope of unintended consequences, or benefits, will be determined during the first year and feedback from an expert advisory panel and a local evaluator panel. As this list is reduced, data sources and collection strategies will also be refined. The primary data sources (i.e., TUP-S, TASP, BHPS, and EIRF-S) and associated methods of data collection can be found in Section A.2.a.

Exhibit 17. Possible Sources and Indicators by Unintended Benefit (Cost)

Unintended Consequence	Indicator	Source
Increase identification and referrals of suicidal adults	Number of trainees who use the training to work with adult at-risk populations	TUP-S Training Core
Diffusion of knowledge to nonparticipants	Trainees who discussed and/or share training materials with others	TUP-S Training Core
Reduce risk of other mental health problems among youth	Number of youth participating in life-skills development trainings	TASP Training Core
Improve care of suicidal adults	Number of adults screened, assessed, with a safety plan, contacted/seen weekly as part of management plan	BHPS SSE Core
Increase early identification of youth with emerging mental health or substance abuse problems	Number of youth who screened positive for other mental health–related issues	EIRF-S
Increase in suicide by alternative means or nonsuicide mortality (e.g., accidents)	Suicide rate by method (firearm vs. other); accidental death rate	CMF
Increase help-seeking behavior, identification, and referrals associated with mental health problems	Proportion of youth receiving mental health services among those with mental illness	NSDUH

The unintended consequence analysis will explore, and when possible quantify, the magnitude of consequences of program implementation beyond those directly related with youth suicide prevention. The analysis will be structured around the following steps: (1) identify potential unintended consequences for each type of prevention strategy implemented by GLS grantees; (2) crosswalk with GLS National Outcomes Evaluation data collection instruments, as well as secondary sources to identify pertinent source of information; and (3) synthetize existing evidence on potential unintended consequences and quantify its relative importance within GLS context in the cases where related information is collected.

Further analysis will be carried out for the subset of potential unintended consequences for which information related to their possible magnitude in the GLS context, in particular, is available. In those cases, we will rely on descriptive statistics to provide precise characterization of the potential magnitude of the unintended consequence (e.g., number of adults identified as at risk of suicide and receiving adequate treatment by behavioral health providers partnering with GLS grantees).

3. Display of Expiration Date

All data collection instruments will display the expiration date of OMB approval.

3. Exceptions to the Certification Statement

This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.

1 The evaluation as designed includes four stages (context, content, process, and impact) each of which is hinged to the fundable activities of the grantees, the research questions outlined in the evaluation statement of work, and the state of the knowledge base in the field of suicide prevention. As such, while the evaluation design does not currently include rigorous impact assessment, it does include the comparative assessment of proximal outcomes as a part of the impact stage. Hereafter, the impact stage is used as an umbrella term to cover evaluation protocols designed and implemented to understand the outcomes of the program. 

2 (Gould, Cross, Pisani, Munfakh, & Kleinman, 2013; Gould & Kalafat, 2009; Gould, Kalafat, Munfakh, & Kleinman, 2007; Gould, Munfakh, Kleinman, & Lake, 2012; Kalafat, Gould, Munfakh, & Kleinman, 2007; King, Nurcombe, Bickman, Hides, & Reid, 2003; Knox, Kemp, McKeon, & Katz, 2012; Mishara et al., 2007a & 2007b)

SS Part A Garrett Lee Smith National Outcomes Evaluation 20