SS Part B 09.07.15

National Outcomes Evaluation

Garrett Lee Smith (GLS) Youth Suicide

Prevention and Early Intervention Program

Supporting Statement

2. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The respondent universe and sampling methods are described below for the following data collection activities: TUP-S, BHPS, and SMSS. The following data collection activities are reports on grant activities or existing data abstractions required from every grantee, so no sampling is required: PSI, TASP, EIRF-I, EIRF-S, and SBHF. Respondents to these activities will be program staff and/or project evaluators. Recent response rates to appropriate activities, along with psychometric analyses, are presented in Exhibit 7 in Section B.4.

Prevention Strategies Inventory (PSI): the PSI is administered on a quarterly basis over the course of the grant period, in the month following the end of the FY quarter. Each grantee designates a program staff respondent. Sampling is not required for the PSI.

Training Utilization and Preservation Survey (TUP-S): the TUP-S will be conducted with a random sample of adults participating in State/Tribal and Campus trainings. Respondents to the TUP-S at 3 months are asked to consent to be contacted at 6 months. The sampling frame is constructed on an ongoing basis as trainings are implemented and grantees administer consents to participate to each adult training participant. Since the final composition and size of the sampling frame is unknown in advance, systematic random sample is used to select a probabilistic sample of training participants who consent to participate. The sampling rate is set by ICF staff and reviewed annually to ensure target sample size per stratum given the anticipated number of trainees and consent rate.

Power analyses were performed based on target sample size. It is estimated that it will be necessary to sample close to 6,000 consents from adult participants in State/Tribal Program-sponsored trainings to achieve the desired sample size. This is based on an initial response rate of 35% among those who consent to participate, and an attrition of 70% in the subsequent follow-up (85% consent to 6-month follow up, 35% of those who consent complete the survey). Approximately 2,000 State/Tribal grantee trainees will participate in the TUP-S at 3 months and 600 trainees will participate at 6 months annually.

The main quantities of interest relate to suicide prevention behavior following training participation (particularly, identification of youth at risk of suicide) and the association of this behavior with training and participant characteristics. The power analysis (Exhibit 1), is focused on the difference between two groups (e.g., participants in two types of training) involving different fractions of the sample (50%, 25%, and 10%) called domains. ICF computed the minimum detectable difference in a proportion (such as the proportion of respondents who identified at least one youth in the past 3 months) assuming the maximum possible variability (i.e., 0.25) and a correlation of 0.5 over repeated measurements. The absolute difference in the proportions (expressed in percentage points), rather than relative difference or difference in odds, was used to facilitate interpretation.

State/Tribal target samples allow for detection of differences smaller than 10 percentage points (pp) even in small domains when combining information across the years, and in medium domains when focusing in a single year. Differences in the 10–15 pp range are detectable in small domains in a single year. Differences in the 10–15 pp range are detectable in large domains in a single year. Inference conditional on the particular set of sites and trainings in a given period is at least as precise as in Exhibit 1. Yet sites and trainings can be thought of as samples of a hypothetical larger population. This would introduce clustering; however, recent empirical work showed no evidence of significant clustering at either level, once basic predictors were taking into account (Condron et al., 2014).

Exhibit 1. Minimum Detectable Difference With 80% Power at 5% Significance Level

Administration	Target Sample Size per Year	Domain1	Minimum Detectable Difference (in percentage points)
			Single Year	Over 3 Years
3 months	2,000	Large	6.3	3.6
		Medium	8.9	5.1
		Small	14.0	8.1
6 months	600	Large	5.7	3.3
		Medium	8.1	4.7
		Small	12.8	7.4

Consistent with previous administrations, approximately 500 Campus grantee trainees will participate in the TUP-S Campus Version 3 months after training. The TUP-S Campus will be administered with Cohort 7 and 8 Campuses in OMB Year 1 only in order to complete data collection under the previously approved protocol. Exhibit 2 describes the power calculation.

Exhibit 2. Minimum Detectable Difference With 80% Power at 5% Significance Level

Administration	Target Sample Size per Year	Domain2	Minimum Detectable Difference (in percentage points)
			Single Year
3 months	500	Large	12.5
		Medium	17.7
		Small	28.0

TUP-S Randomized Controlled Trial (RCT) Versions: all adults who participate in RCTT training activities and consent to be contacted will be eligible for the TUP-S RCT. ICF will identify the number of State/Tribal grantees implementing gatekeeper trainings, focusing primarily on sites implementing at least 10 trainings per year. A subset of 10 grantees will participate in the Training Study RCT. All trainees from RCT grantees who consent to be contacted will be eligible.

Two levels of randomization will take place as part of the factorial RCT design, first into baseline training groups with and without role-play, and 2^nd into groups with or without a booster after training, as outlined in Exhibits 3 and 4 below.

Exhibit 3. TUP-S RCT First Level of Randomization for Enhanced Study

Activity		Booster (6 months)		Total
		Yes	No
Role-play (Baseline)	Yes	333	333	666
	No	333	333	666
TOTAL		666	666	1,332

The second stage of randomization will take place following the first set of trainings.

Exhibit 4. TUP-S RCT Second Level of Randomization for Enhanced Study

Training Condition	Number to Recruit
Brief gatekeeper training only	333
Brief gatekeeper training + role-play	333
Brief gatekeeper training + booster	333
Brief gatekeeper training + role-play+ booster	333
Total	1,332

Exhibit 5 presents the sample size required to detect a difference of 10 percentage points associated with each of the tested factors; that is, trainees participating in trainings incorporating a role-playing exercise (as opposed to trainings that do not incorporate such exercise) and participants exposed to a booster intervention after the training (as opposed to those who are not).

Exhibit 5. TUP-S RCT Estimated Number of Cases Needed to Detect a Difference of 10 Percentage Points With 80% Power at 5% Significance Level

Indicator (Assumed Baseline)	Detectable Change	Effective Size	Number to Recruit to Participate
Difference (between arms) in the proportion of participants who identify at least one youth at risk, refer the youth to support, and is aware whether the youth received the support at 12 months (baseline 37%)	10 pp	366 trainees per arm in the last administration	666 trainees per arm at baseline (assuming 45% attrition)

The Training Study RCT will rely on a factorial randomized design to assess, in real-life settings, the impact of two factors in enhancing the effectiveness of gatekeeper training: the incorporation of active learning techniques (in particular, role-playing exercise), and the use of a booster session 3 to 6 months following participation the initial training. Due to randomization, a comparison of proportions between arms (e.g., the proportion of participants who identify at least one youth at risk, refer the youth to support, and are aware whether the youth received the support) may be directly informative of difference in effectiveness. Additionally, regression analysis will be performed to identify variations in effectiveness by trainee baseline characteristics, such as trainee role, setting in which they typically interact with youth, and typical interaction time with youth.

Difference in nonresponse patterns between arms is a common challenge encountered in a randomized study setting. Particularly challenging are situations where nonresponse is associated with the value of the unobserved outcome. In the present application, however, we do not anticipate the type of training assigned to predict response rates. Several methods are available to handle missing data if the data are missing completely at random or missing at random after baseline characteristics or previous values of the outcome are controlled for. Compliance, on the other hand, may plausibly be predicted by assignment (e.g., in the case of the booster intervention). Although, the so-called “intention-to-treat” analysis will still be informative, instrumental variable estimation will be implemented to estimate effectiveness associated with actual participation (as opposed to assignment to a type of training).

Behavior Health Provider Survey (BHPS): the BHPS baseline will be administered to 1 to 10 administrators/supervisors from the behavioral health provider organization partnering with all newly funded State/Tribal grantees in Year 1 of their grant; follow-up BHPSs will be administered to the same behavioral health providers annually for the remainder of the grant period. Cohort 9 State/Tribal grantees will only collect BHPS data for 4 years of their 5-year grant cycle. They will implement the baseline BHPS in Year 2 of their grant, as the data collection protocol will be submitted for OMB approval in Year 1. Because the BHPS is administered to all behavioral health providers partnering with GLS State/Tribal grantees, there is no need for sampling. Rather, ICF will work with the grantee and administrators within the organizations to determine the most appropriate respondent(s) to represent each organization. Respondents will be selected based on their ability to report on suicide safer care practices within their organization.

Referral Network Survey (RNS): the RNS will be administered to staff members from referral network organizations associated with Cohort 8 State/Tribal grantees. Grantees will provide contact information for up to 5 organizations in the network. ICF staff members will contact organization to determine the entire referral network for the RNS.

Short Message Service Survey (SMSS): the SMSS will be conducted with Cohort 7 Campus grantees as the second and final administration to fulfill data collection requirements. In line with the previous administration, ICF will target a convenience sample of 100 students per campus.

2. Information Collection Procedures

PSI: one month after the end of each FY quarter, the grantee receives a PSI password via e-mail and uses the password to log in to the survey on the SPDC. The respondent enters program strategy and budget information and must finalize the submission by the end of the administration period, which lasts for 15 business days. All of the program strategies are reviewed each quarter by the PSI instrument lead to ensure data quality. PSI respondents are provided with technical assistance via e-mail (e.g., help email) for any questions on how to categorize or enter prevention strategies implemented through their GLS program.

TASP: Grantees will receive a username and password to submit TASP data to the SPDC via a Web-based form. ICF will train grantees on entering the data and will monitor participation.

TUP-S and TUP-S Campus: during State/Tribal and Campus training events, potential participants will be asked to complete a consent-to-contact form indicating their willingness to be contacted to participate in the TUP-S. Grantee program staff will gather consent-to-contact information before training event as part of their registration process, or prior to implementing activities at the training. The consent to contact forms will be shared with ICF via the SPDC, e-mail, or mail within 2 weeks of the event to establish the follow-up survey date and provide time to process contact information. Consent to contact is gathered via hardcopy or electronic form. The electronic form will be also offered to grantees to facilitate data collection from trainees participating in Webinars and online trainings and to streamline submission processes. Once consent to contact has been received, ICF will contact a random sample of key informants via telephone 3 months following the training event to introduce the study, request participation, and to schedule an appointment for administration of the interview. Respondents will provide verbal consent before administration. At the end of the 3-month interview, State/Tribal trainee respondents will be asked for consent for recontact in another 3 months (6-month survey). ICF will then contact a random sample of those individuals to request their participation. The TUP-S will be implemented using CATI technology via the ICF call center. All of the CATI stations are equipped with predictive dialing capabilities.

TUP-S RCT (Training Study RCT): ICF will contact prospective participants in the RCT prior to their training for consent to participate in all TUP-S RCT administrations (baseline, 3-, 6-, and 12-months). Prior to beginning each administration, participants are re-consented. All four versions of the TUP-S for the enhanced RCT study will be CATI surveys administered by the ICF call center, and data will be uploaded to the SPDC for processing and analysis. Once sites have been selected, grantees will schedule and recruit participants for one of two brief gatekeeper trainings and share a consent-to-contact form with prospective participants at registration. At the first stage of randomization, ICF staff will assist grantees in randomizing prospective trainees into one of two training groups: (1) brief gatekeeper training or (2) brief gatekeeper training with role-play. Participants will be told that they will take part in a brief gatekeeper training, but will not know that the two trainings offered will have different components. The participants’ group assignment will be tracked in a database maintained by ICF.

The booster activity will be scheduled at two time points in the months following the first training: at 2 and 4 months posttraining. The second stage of randomization will take place following the first set of trainings. ICF will help grantees randomly assign half of the brief gatekeeper-only trainees and half of the brief gatekeeper plus role-play trainees to participate in the remaining training groups: (1) a booster training activity or (2) no booster activity. The grantee will contact participants assigned to the booster activity group about participating in the activity following assignment. Participants’ group assignment will be tracked in a database maintained by ICF. Recruitment will begin after OMB clearance is received with a staggered roll-out during Years 2 and 3 of the evaluation, until the baseline recruitment goal of 1,332 participants has been met.

Early Identification, Referral, and Follow-up Individual Form (EIRF-I): Grantee staff upload EIRF-I data each quarter to the SPDC. Initial follow-up information (whether or not a service was received after referral) is obtained within 3 months. Details from the second follow-up appointment (e.g., type of services received) also will be collected for the 3-month period. Data are extracted from case records or other existing data sources, including any organizational staff, community members, or family members who make a mental health identification and referral. For grantees that do not have access to an existing tracking system, they should contact their TAL, prevention specialist, and SAMHSA Government Project Officer (GPO) to discuss approaches for adequately tracking and monitoring youth identified and referred for services.

EIRF Screening Form (EIRF-S): The grantee will submit EIRF–S forms each quarter. EIRF-S forms are completed once per each implementation of a screening tool in a group setting, once per month for clinical screenings, and once per month for one-on-one screenings. For each screening event where multiple youth are screened at a given time, one EIRF–S should be completed for the event. For one-on-one screenings in a clinical or other setting, one aggregated EIRF–S is completed per month to reflect screening outcomes of all youth screened during the month. Grantees develop systems locally to gather identification and referral data, including extracting data from existing electronic health records (EHRs) or forms; grantee program staff enter EIRF data into a Web-based survey on the SPDC on an ongoing basis throughout their grant period.

Referral Network Survey (RNS): the RNS is administered to referral network organizations associated with State/Tribal Program grantees. Staff members at each organization complete the survey. Grantees provide contact information up to 5 organizations in the area. ICF staff members make preliminary phone calls to request information on other organizations in the referral network. ICF will make preliminary telephone calls to determine the entire referral network for the county or region. Evaluation staff will contact organizations and/or agencies that form the referral network to complete the survey online. The RNS will be administered in year 1 of OMB approval with Cohort 8 State/Tribal grantees to complete the second and final administration of the instrument.

BHPS: ICF will collect contact information for the behavioral health partner providers from State/Tribal grantee staff each year and will send an e-mail with a link to the survey to the provider. Reminder e-mails and calls will be made during the administration window to increase provider participation. Providers will access the Web survey link via e-mail and complete the survey on the SPDC. In order to capture the potentially lengthy set of NPIs associated within any provider organizations needed to link to the Medicaid claims data for analyses, the BHPS will include a link so that providers can upload a file containing NPIs rather than entering them into the survey. The baseline BHPS will be administered 1 to 10 administrators from the behavioral health provider organization partnering with all newly funded State/Tribal grantees in Year 1 of their grant; follow-up BHPSs will be administered to the behavioral health provider annually for the remainder of the grant period. Cohort 9 State/Tribal grantees will only collect BHPS data for 4 years of their 5-year grant cycle. They will implement the baseline BHPS in Year 2 of their grant, as the data collection protocol will be submitted for OMB approval in Year 1.

SMSS: the SMSS will be administered via mobile telephone text message to a sample of 100 students from each Cohort 7 campus as the second and final administration for the cohort. Students will be recruited for the SMSS through an in-person intercept method or potentially through an e-mail list. Through the intercept recruitment method, evaluation staff members travel to the campus and approach students in common areas to ask whether they are willing to participate in the survey. For the e-mail list recruitment method, the ICF evaluation team either obtains e-mail addresses for students from campus administration or campus staff set up a student e-mail list; ICF then sends e-mail invitations to students about survey and provides a link to enter their mobile telephone numbers to participate. Following recruitment, the survey is sent via text message by ICF staff. Respondents will receive an initial text message to participate in a text survey about suicide prevention. Participants responding “yes,” will receive 2-3 messages containing consent language, including background information about the study, risks and benefits, as well as information to contact the NSPL if they require help. After the consent process, students will have the option to select “yes” to continue to the survey.

Student Behavioral Health Form (SBHF): Grantee program staff (e.g., evaluator, project director) from each Campus grantee will participate in the SBHF. Campus program staff members work with relevant departments on campus to obtain information and submit electronically annually each spring. At baseline, campus program staff submits information for the five most recent years, including the current year. During Years 2 through 3 of the grant, staff members submit current-year information only. SBHF data are submitted using a Web-based form on the SPDC annually.

3. Methods to Maximize Response Rates

Participation in the NOE is a requirement of the GLS Suicide Prevention Program. Therefore, completion of the PSI, EIRF-I, EIRF-S, SBHF, and TASP (for sites providing training) by program staff will be a requirement. However, the NOE team has taken a number of steps to minimize the burden on local programs to ensure that completion is timely. These steps include developing a Web-based data collection system, using updated technology, and providing training and technical assistance (TTA) to grantees. The NOE team also will provide TTA to maximize response rates for the other data collection activities by hosting Web trainings, distributing procedures manuals, and conducting onsite training visits for the State/Tribal grantees as appropriate.

TUP-S: methods to be used to maximize response rates for telephone surveys include obtaining buy-in from key program stakeholders, providing flexibility in scheduling, and conducting follow-up phone calls and emails to nonresponders. In addition, local program staff will be utilized to obtain contact information for respondents, which will result in more accurate information, thus increasing response rates. If any identified respondents are nonresponsive, the NOE team will request that local program staff identify replacement respondents as appropriate.

RNS and BHPS: procedures to maximize participation in the RNS and BHPS include using a Web-based data collection system for administration of the instruments, as well as providing TTA for completing the survey. For the RNS, grantees will provide contact information for up to 5 organizations in the area and NOE team members will gather information on other organizations in the referral network. For the BHPS, the grantee will identify their primary administrative contact at the partnering behavioral health provider, who would serve as a respondent and can also suggest other potentially informative respondents. For both surveys, local program staff will also be utilized to obtain contact information for respondents. Reminder e-mails and calls will be made during the administration window to increase provider participation.

SMSS: response rates will be augmented by conducting the survey via text message, allowing students to complete the survey at a time convenient for them and via technology with which they are familiar.

4. Tests of Procedures

Drawing on a 9-year experience collecting data through the evaluation of the GLS program and findings from the evaluation, improvements have been made to the administration protocols and content of data collection instruments. As new measures were developed, standard instrument development procedures, including review of the literature, item development, and content review by experts in the field were used. All instruments underwent cognitive and/or pilot testing, and/or expert review. These procedures were used to enhance question accuracy and determine administration times. In addition, Web-enabled and SMS instruments will undergo usability testing prior to fielding. Usability testing refers to pilot testing of the interface for administering questionnaires to determine the most efficient and understandable presentation. Typically, this is completed with a prototype and modifications are made before final fielding.

Combined with a review of GLS program foci requirements and cross-site evaluation findings to date, a thorough literature review related to suicide prevention training effectiveness (activities, components, and practices); early identification and referral on subsequent care follow-up and adherence; and suicide safer care practices within health care settings was conducted to develop new instruments (TUP-S RCT and BHPS) and inform the revisions of others. In addition, experts in behavioral health and mental health referral networks were consulted in developing the BHPS. Second, drafts of the instruments were developed and reviewed by NOE team members, survey methodologists, representatives from SAMHSA, and content experts in the field of suicide prevention. Item analyses were conducted across instruments to be sure that key critical items were assessed similarly across all questionnaires. To enhance question accuracy and determine administration time, instruments underwent cognitive and/or pilot testing or expert review.

Exhibit 7 below outlines the response rates and psychometric analyses associated with NOE data collection activities, as well as revisions to existing protocols to maximize response rates.

Exhibit 7. Data Collection Activity Revisions and Response Rates

Instrument	Response Rate &Psychometric Analyses Information3	Revisions to Proposed Protocol 4
PSI - Revised	Among currently funded grantees, 94.7% of Campuses and 91.7% of States and Tribes participated in the last PSI and participated. Psychometric analyses are not appropriate.	No revisions related to response rates proposed; content changes only.
TUP-S Revised	Initial TUP-S response rates were 35% among a random sample of individuals who provided consent to contact. Of those, 85% consent to be contacted at the 6-month follow-up. Initial findings indicate that 35% of respondents who consent to participate at 6 months complete the survey. Psychometric analyses are not appropriate.	An online version of the TUP-S consent to contact form will be available for participants and grantees to facilitate improved response rates. The TUP-S uses random and probabilistic sampling.
TUP-S RCT New	Response rates to the TUP-S baseline and 12 month have not yet been established. For the RCT, the TUP-S will be administered to all training participants (rather than a random sample) until the desired sample size is achieved (1,332). Psychometric analyses are not appropriate for the TUP-S RCT Versions.	The factorial RCT uses the same recruitment procedures as the TUP-S. Baseline occurs prior to training. Grantees will recruit participants at registration for trainings or provide the mobile-friendly link to the online consent-to-contact form. Collected forms must be sent to ICF to allow for baseline to occur 2 weeks prior to training.
BHPS New	The BHPS is a new data collection activity; thus, response rates have not been established. The survey was adapted from the Zero Suicide Organizational Self-Study: http://zerosuicide.sprc.org/resources/zero-suicide-organizational-self-study. Some items were also taken from the 2014 National Mental Health Services Survey, available here: http://info.nmhss.org/	The BHPS will be implemented as a Web-based survey of behavioral health partner providers to State/Tribal grantees.
RNS	Previous administrations of the RNS with cohorts 1-6 have yielded between 44-81% agency response rates. 78.2% of cohort 7 grantees participated in the last administration of the RNS. The wave yielded between 8–100% agency response rates. Previous administrations of the RNS with cohorts 1–6 have yielded between 44-81% agency response rates. Psychometric analyses are not appropriate for the RNS.	N/A
EIRF-I & EIRF-S Revised	The EIRF-I and EIRF-S do not have identified samples and therefore response rate information is not applicable; however, we monitor the participation of grantees in each activity. Overall 48.2% of currently funded grantees participate in the EIRF (87% of cohort 7, 57% of cohort 8, and 12% of cohort 9 grantees are participating in the EIRF). Psychometric analyses are not appropriate.	The EIRF-I and EIRF-S will continue to collect information about youth identified at-risk by gatekeepers and/or via screening tools. Initial referral follow-up information and details about second appointments should be obtained within 3 months. No revisions related to response rates are proposed—only content changes.
SBHF Revised	Almost all grantees participate in the SBHF/MIS, among currently funded grantees. 98.3% of campuses participated in the most recent MIS administration in FY 2014 (cohort 6 97% and cohort 7 100%). Psychometric analyses are not appropriate.	The SBHF will request information about the number of student suicide attempts and completions; service use, referral, and identifications; and behavioral services on campus. Campuses will work more closely with counseling centers to complete the form so that all sites will be able to respond to procedural questions regardless of the availability of specific counts.
SMSS	Almost all Campus grantees participate in the SMSS. Between spring/summer of 2014 and November 2014, there were 2,814 SMSS responses from 32 campuses. Among all students contacted about the survey, 26% participated. Of the students who started the survey, almost 93% fully completed it.	N/A

5. Statistical Consultants

ICF has full responsibility for the development of the overall statistical design, and assumes oversight responsibility for data collection and analysis. Training, TA, and monitoring of data collection will be provided by the NOE team. The individuals responsible for overseeing data collection and analysis are:

Christine M. Walrath, PhD

ICF Macro, Inc.

40 Wall Street, 34^th Floor

New York, NY 10005

Phone: (212) 941-5555

E-mail: [email protected]

The following individuals will serve as statistical consultants to this project:

Christine M. Walrath, PhD

ICF Macro, Inc.

New York, NY 10005

Robert Stephens, PhD

ICF Macro, Inc.

3 Corporate Square, Suite 370

Atlanta, GA 30329

Phone: (404) 321-3211

E-mail: [email protected]

Lucas Godoy Garraza, PhD, Statistician

Teleworks—Home Office

E-mail: [email protected]

Wendy Cross, PhD

University of Rochester

206 Wallis Hall, P.O. Box 270026

Rochester, NY 14627

E-mail: [email protected]

The agency staff person responsible for receiving and approving contract deliverables is:

Anne Mathews-Younes, EdD, Director and COR

Division of Prevention, Traumatic Stress, and Special Programs

Center for Mental Health Services, SAMHSA

1 Choke Cherry Road, Room 6-1093

Rockville, MD 20857

Phone: (240) 276-1860

E-mail: [email protected]

References

American Foundation for Suicide Prevention. (2010). Facts and figures. Retrieved April 15, 2014, from http://www.afsp.org/understanding-suicide/facts-and-figures

Centers for Disease Control and Prevention. (2010). 10 leading causes of death by age group, United States—2010. Retrieved April 15, 2014, from http://www.cdc.gov/injury/wisqars/pdf/10LCID_All_Deaths_By_Age_Group_2010-a.pdf

———. (2013). Suicide among adults aged 35–64 years—United States, 1999–2010. Morbidity and Mortality Weekly Report, 62(17), 321–325.

———. (2012). Suicide facts at a glance. Retrieved April 15, 2014, from http://www.cdc.gov/violenceprevention/pdf/Suicide-DataSheet-a.pdf

Centers for Disease Control and Prevention, National Center for Health Statistics. (2013). FastStats. Suicide and self-inflicted injury. Retrieved May 1, 2014, from http://www.cdc.gov/nchs/fastats/suicide.htm

Center for Native American Youth at the Aspen Institute. (n.d.). Native American youth 101: Information on the historical context and current status of Indian Country and Native American youth fact sheet. Retrieved June 12, 2014, from http://www.aspeninstitute.org/sites/default/files/content/upload/Native%20American%20Youth%20101_higres.pdf

Condron, D. S., Godoy-Garraza, L., Walrath, C., Mckeon, R., & Heilbron, N. (2014). Referral and service receipt for youths identified as being at risk for suicide: Relationships between training type, professional role, and referral behavior in school-based trainees. Suicide and Life-Threatening Behavior. doi:10.1111/sltb.12142

Crosby, A. E., Han, B., Ortega, L. A., Parks, S. E., & Gfoerer, J. (2011). Suicidal thoughts and behaviors among adults aged ≥18 years—United States, 2008–2009. MMWR Surveillance Summaries, 60(13), 1–22. Retrieved March 10, 2014, from http://www.cdc.gov/mmwr/preview/mmwrhtml/ss6013a1.htm

Daniels, A. S., Shaul, J. A., Greenberg, P., Cleary, P. D. & Maruish, M. E. (2004). The use of psychological testing for treatment planning and outcomes assessment: Volume 3: Instruments for adults (3rd ed.). Mahwah, NJ: Lawrence Erlbaum Associates Publishers.

DiMatteo, M. R., Sherbourne, C. D., Hays, R. D., Ordway, L., Kravitz, R. L., McGlynn, E. A., et al. (1993). Physicians’ characteristics influence patients’ adherence to medical treatment: Results from the Medical Outcomes Study. Health Psychology, 12(2), 93–102.

Garlow, S. J., Rosenberg, J., & Moore, J. D. (2008). Depression, desperation, and suicidal ideation in college students: Results from the American Foundation for Suicidal Screening Project at Emory University. Depression and Anxiety, 25, 482–488.

Goldston, D. B., Molock, S. D., Whitbeck, L. B., Murakami, J. L., Zayas, L. H., & Hall, G. C. (2008). Cultural considerations in adolescent suicide prevention and psychosocial treatment. American Psychologist, 63(1), 14–31.

Goldston, D., Walrath, C., McKeon, R., Puddy, R., Lubell, K., Potter, L., et al. (2010). The Garrett Lee Smith Memorial Suicide Prevention program. Suicide and Life-Threatening Behavior, 40(3), 245–256.

Gould, M. S., Cross, W., Pisani, A. R., Munfakh, J. L., & Kleinman, M. (2013). Impact of applied suicide intervention skills training on the National Suicide Prevention Lifeline. Suicide and Life Threatening Behavior, 43(6), 676–691.

Gould, M. S., & Kalafat, J. (2009). Role of crisis hotlines in suicide prevention. In D. Wasserman & C. Wasserman (Eds.), The Oxford textbook of suicidology—The five continents perspective (pp. 459–462). Oxford, UK: Oxford University Press.

Gould, M. S., Kalafat, J., Munfakh, J. L. H., & Kleinman, M. (2007). An evaluation of crisis hotline outcomes, Part II: Suicidal callers. Suicide and Life Threatening Behavior, 37(3), 338–352.

Gould, M. S., Munfakh, J. L. H., Kleinman, M., & Lake, A. M. (2012). National Suicide Prevention Lifeline: Enhancing mental health care for suicidal individuals and other people in crisis. Suicide and Life-Threatening Behavior, 42(1), 22–35.

Heilbron, N., Goldston, D., Walrath, C., Rodi, M., & McKeon, R. (2013). Suicide risk protocols: Addressing the needs of high risk youths identified through suicide prevention efforts and in clinical settings. Suicide and Life-Threatening Behavior, 43(2), 150–160.

ICF International. (2013). Garrett Lee Smith Youth Suicide Prevention and Early Intervention Cross-site Evaluation: Fiscal Year 2013 annual report (Report to the Substance Abuse and Mental Health Services Administration). Atlanta, GA: Author.

———. (2014). Garrett Lee Smith Youth Suicide Prevention and Early Intervention Cross-site Evaluation: Fiscal Year 2014 annual report (Report to the Substance Abuse and Mental Health Services Administration). Atlanta, GA: Author.

Isaac, M., Elias, B., Katz, L. Y., Belik, S., Deane, F. P., Enns, M. W., et al. (2009). Gatekeeper training as a preventative intervention for suicide: A systematic review. Canadian Journal of Psychiatry-Revue Canadienne de Psychiatrie, 54, 260–268.

Kazdin, A., & Wassell, G. (1999). Barriers to treatment participation and therapeutic change among children referred for conduct disorder. Journal of Clinical Child Psychology, 28, 160–172.

Keller, D. P., Schut, L. J., Puddy, R. W., Williams, L., Stephens, R. L., McKeon, R., et. al. (2009). Tennessee lives count: Statewide gatekeeper training for youth suicide prevention. Professional Psychology: Research and Practice, 40(2), 126–133.

King, K. A. & Smith, J. (2000). Project Soar: A training program to increase school counselors’ knowledge and confidence regarding suicide prevention and intervention. Journal of School Health, 70(10), 402–407.

King, R., Nurcombe, R., Bickman, L., Hides, L., & Reid, W. (2003). Telephone counseling for adolescent suicide prevention: Changes in suicidality and mental state from beginning to end of a counseling session. Suicide and Life Threatening Behavior, 33(4), 400–411.

King, K. A., Vidourek, R. A., & Strader, J. L. (2008). University students’ perceived self-efficacy in identifying suicidal warning signs and helping suicidal friends find campus intervention resources. Suicide and Life Threatening Behavior, 38(5), 608–617.

Knox, K. L., Kemp, J., McKeon, R., & Katz, I. R. (2012). Implementation and early utilization of a suicide hotline for veterans. American Journal of Public Health, 102(Suppl. 1), S29–S32.

Lynch, F. L. (2014). Population health outcome models in suicide prevention policy. American Journal of Preventive Medicine, 47(3S2), S137–S143.

Matthieu, M. M., Cross, W., Batres, A. R., Flora, C. M., & Knox, K. L. (2008). Evaluation of gatekeeper training for suicide prevention in veterans. Archives of Suicide Research, 12(2), 148–154.

Michelmore, L., & Hindley, P. (2012). Help-seeking for suicidal thoughts and self-harm in young people: A systematic review. Suicide and Life Threatening Behavior, 42(5), 507–524.

Mishara, B. L., Chagnon, F., Daigle, M., Balan, M., Raymond, S., Marcoux, I., et al. (2007a). Which helper behaviors and intervention styles are related to better short term outcomes in telephone crisis intervention? Results from a silent monitoring study of calls to the U.S. 1-800-SUICIDE Network. Suicide and Life Threatening Behavior, 37(3), 291–307.

———. (2007b). Comparing models of helper behavior to actual practice in telephone crisis intervention: A silent monitoring study of calls to the U.S. 1-800-SUICIDE Network. Suicide and Life Threatening Behavior, 37(3), 308–321.

Montgomery, A. A., Peters, T. J., & Little, P. (2003). Design, analysis, and presentation of factorial randomized controlled trials. BMC Medical Research Methodology, 3–26.

National Action Alliance for Suicide Prevention. (2014). A prioritized research agenda for suicide prevention: An action plan to save lives. Rockville, MD: National Institute of Mental Health and the Research Prioritization Task Force.

———. (2011). Continuing contact after care. Washington, DC: Author. Retrieved January 15, 2015, from http://zerosuicide.actionallianceforsuicideprevention.org/continuing-contact-after-care.

National Center for Health Statistics Vital Statistics System. (2010). Alaska all races, both sexes, before age 65. Retrieved on December 15, 2014, from http://www.cdc.gov/injury/wisqars/pdf/10lcid_violence_related_injury_deaths_2010-a.pdf

New Freedom Commission on Mental Health. (2003). Achieving the promise: Transforming mental health care in America: Final report (DHHS Pub. No. SMA-03-3832). Rockville, MD: Author.

Parkerson G. R. Jr., Broadhead W. E., Tse, C-K. J. (1996). Anxiety and Depression Symptom Identification Using the Duke Health Profile. Journal of Clinical Epidemiology, 49(1), 85-93.

Piacentini, J., Rotheram-Borus, M. J., Gillis, J. R., Graae, F., Trautman, P., Garcia-Leeds, C., et al. (1995). Demographic predictors of treatment attendance among adolescent suicide attempts. Journal of Consulting and Clinical Psychology, 63, 469–473.

Quinnett, P. (1995). Question, persuade, refer: Ask a question, save a life [Booklet]. Spokane, WA: QPR Institute.

Reis, C., & Cornell, D. (2008). An evaluation of suicide gatekeeper training for school counselors and teachers. Professional School Counselling, 11(6), 386–394.

Rodi, M. S., Godoy-Garraza, L., Walrath, C., Stephens, R., Condron, D. S. Hicks, B. B., et al. (2012, January). Referral patterns for youth identified at risk for suicide by trained gatekeepers. Crisis, 33(2), 113–119.

Rotheram-Borus, M. J., Piacentini, J., Cantwell, C., Belin, T. R., & Song, J. (2000). The 18-month impact of an emergency room intervention for adolescent female suicide attempters. Journal of Consulting and Clinical Psychology, 68(6), 1081–1093.

Spirito, A., Boergers, J., Donaldson, D., Bishop, D., & Lewander, W. (2002). An intervention trial to improve adherence to community treatment by adolescents after a suicide attempt. Journal of the American Academy of Child & Adolescent Psychiatry, 41(4), 435–442.

Spirito, A., Plummer, B., Gispert, M., Levy, S., Kurkjian, J., Lewander, W., et al. (1992). Adolescent suicide attempts: Outcomes at follow-up. American Journal of Orthopsychiatry, 62(3), 464–468.

Substance Abuse and Mental Health Services Administration. (2011). Leading change: A plan for SAMHSA’s roles and actions 2011–2014—Executive summary and introduction (HHS Publication No. (SMA) 11-4629). Rockville, MD: Author.

———. (2014). National Mental Health Services Survey (N-MHSS): 2010. Data on mental health treatment facilities. BHSIS Series S-69, HHS Publication No. (SMA) 14–4837. Rockville, MD: Author.

———. (2005). Results from the 2004 National Survey on Drug Use and Health: National findings (DHHS Publication No. SMA 05-4062, NSDUH Series: H-28). Rockville, MD: U.S. Department of Health and Human Services.

———. (2013). Results from the 2012 National Survey on Drug Use and Health: Mental health findings. NSDUH Series H-47, HHS Publication No. (SMA) 13–4805. Rockville, MD: Author. Available at http://www.samhsa.gov/data/NSDUH/2k12MH_FindingsandDetTables/2K12MHF/NSDUHmhfr2012.htm#sec3–1.

Suicide Prevention Resource Center. (2015). Recommendations for: Youth suicide prevention training for early identification and referral (gatekeeper training). Retrieved January 15, 2014, from http://www.sprc.org/sites/sprc.org/files/library/RecommendationsGatekeeperTraining.pdf

———. (2008). Suicide risk and prevention for lesbian, gay, bisexual, and transgender youth. Newton, MA: Education Development Center, Inc. Retrieved January 7, 2014from http://www.sprc.org/sites/sprc.org/files/library/SPRC_LGBT_Youth.pdf

Tautman, P. D., Stewart, N., & Norishimma, A. (1993). Are adolescent suicide attempters noncompliant with outpatient care? Journal of the American Academy of Child and Adolescent Psychiatry, 32, 89–94.

U.S. Bureau of Labor Statistics. (2014). National Compensation Survey Statistics. Retrieved April 2015 from http://www.bls.gov/ncs/ocs/

U.S. Department of Health and Human Services, Office of the Surgeon General and National Action Alliance for Suicide Prevention. (2012). 2012 national strategy for suicide prevention: Goals and objectives for actions. Washington, DC: U.S. Department of Health and Human Services. Retrieved January 7, 2014, from http://www.surgeongeneral.gov/library/reports/national-strategy-suicide-prevention/full_report-rev.pdf

U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration. (2014a). Garrett Lee Smith Youth Suicide Prevention and Early Intervention Program: Fiscal Year 2014 annual report. Rockville, MD: Author.

———. (2014b). PPHF—2014 cooperative agreements for State-sponsored youth suicide prevention and early intervention (Request for Applications No. SM-14-008). Rockville, MD: Author. Retrieved December 15, 2014, from http://www.samhsa.gov/sites/default/files/grants/pdf/sm-14-008_1.pdf

U.S. Public Health Service. (1999). The Surgeon General’s call to action to prevent suicide. Washington, DC: U.S. Department of Health and Human Services.

Walrath, C., Godoy-Garraza, L., Reid, H., Goldston, D., & McKeon, R. (in press). The impact of the Garrett Lee Smith (GLS) Suicide Prevention Program on suicide mortality. American Journal of Public Health.

Wyman, P. A., Hendricks Brown, C., Inman, J., Cross, W., Schmeelk-Cone, K., Guo, J., et al. (2008). Randomized trial of a gatekeeper program for suicide prevention: 1-year impact on secondary school staff. Journal of Consulting Clinical Psychology, 76, 104–115.

Attachments

1. Garrett Lee Smith Memorial Act

2. NOE Instrument Table

3. Prevention Strategies Inventory (PSI)

1. State/Tribal Version

2. Campus Version

4. Training Activity Summary Page (TASP)

1. State/Tribal Version

2. Campus Version

5. Training Utilization and Preservation Survey (TUP-S)–State/Tribal

1. TUP-S State/Tribal Baseline (RCT)

2. TUP-S State/Tribal 3 Month (Core/RCT)

3. TUP-S State/Tribal 6 Month (Core/RCT)

4. TUP-S State/Tribal 12 Month (RCT)

5. TUP-S Consent to Contact Form (Core)

6. TUP-S Consent to Contact Form (RCT)

6. Training Utilization and Preservation Survey (TUP-S)–Campus

1. TUP-S Campus 3 Month

2. TUP-S Campus Consent to Contact Form

7. Early Intervention, Referral, and Follow-up Individual Form (EIRF-I)

8. Early Intervention, Referral, and Follow-up Screening Form (EIRF-S)

9. Student Behavioral Health Form (SBHF)

10. Behavioral Health Provider Survey (BHPS)

11. Referral Network Survey

12. Short Message Service Survey (SMSS)

13. Training Study Logic Model

14. Continuity of Care (COC) Study Logic Model

15. Suicide Safer Environment (SSE) Study Logic Model

16. SPDC Data Use and Access Agreement

17. Privacy Impact Assessment Form

1 Large domain is defined as comparisons of two groups involving 50% of the sample each. Medium domain is defined as comparisons of two groups involving 25% of the sample each. Small domain is defined as comparisons of two groups involving 10% of the sample each. The 3-month comparison focuses on the cross-sectional proportion at 3 months, or the change in the proportion compared to baseline levels (assuming a correlation of 0.5 over time). The 6-month comparison focuses on the linear rate of change between baseline and 6 months (assuming a constant correlation of 0.5 over time).

2 Large domain is defined as comparisons of two groups involving 50% of the sample each. Medium domain is defined as comparisons of two groups involving 25% of the sample each. Small domain is defined as comparisons of two groups involving 10% of the sample each. The 3-month comparison focuses on the cross-sectional proportion at 3 months, or the change in the proportion compared to baseline levels (assuming a correlation of 0.5 over time). The 6-month comparison focuses on the linear rate of change between baseline and 6 months (assuming a constant correlation of 0.5 over time).

3 Psychometric analyses are indicated for data collection instruments that apply scales that are able to be analyzed. If the activity doesn’t collect scaled data it is not considered appropriate for psychometric analysis and is thereafter indicated as “not appropriate”.

4 The approaches to address response rates and implementation challenges through protocol revision have been included in the last column of the exhibit.

SS Part B Garrett Lee Smith National Outcomes Evaluation 11