The Substance Abuse and Mental Health Services Administration’s (SAMHSA’s) Division of Prevention, Traumatic Stress and Special Programs of the Center for Mental Health Services (CMHS) is requesting clearance for data collection associated with the cross-site evaluation of the Garrett Lee Smith (GLS) Memorial Youth Suicide Prevention and Early Intervention Program—the GLS State/Tribal Suicide Prevention Program (State/Tribal Suicide Prevention Program) and the GLS Campus Suicide Prevention Program (Campus Suicide Prevention Program). The Garrett Lee Smith Memorial Act (GLSMA), passed by Congress in October 2004, was the first legislation to provide funding specifically for State/Tribal and Campus Suicide Prevention programs. This legislation sets aside funding for states, tribes, and institutions of higher education to develop, evaluate, and improve early intervention and suicide prevention programs, and mandates that the effectiveness of programs be evaluated and reported to Congress (see Attachment A).
The currently approved cross-site evaluation of the GLS Suicide Prevention Program is for the data collection associated with 8 instruments collected by State/Tribal grantees, 7 instruments collected by Campus grantees, and 3 instruments collected by a subset of Campus grantees. The proposed revised data collection protocol includes the removal of 1 instrument collected by States/Tribes, 2 instruments collected by Campuses, and all instruments associated with the enhanced evaluation. The revised protocol includes the addition of 5 instruments collected by States/Tribes and 3 instruments collected by Campuses.
To date, SAMHSA has awarded 147 State/Tribal grants and 153 Campus grants under the GLSMA. The cross-site evaluation of the GLS Suicide Prevention Program was designed to evaluate the effectiveness of suicide prevention activities across multiple sites and to report those findings to Congress. While the desired long-term outcome of suicide prevention activities is a reduction in suicide attempts and deaths by suicide, there are potential intermediary variables that must be adequately and robustly evaluated prior to the evaluation of suicidal behavior itself. Complex conceptual models that include intermediary pathways of effect, such as those that underpin suicide prevention programs, must be evaluated using a staged framework that allows for the assessment of process, mediating, and long-term outcomes (i.e., potential mediating variables). For example, many suicide prevention programs currently do not have information on whether youth identified as at risk for suicide are able to access treatment—an intermediate variable that requires investigation. Using components designed to capture process, proximal, and intermediate outcomes, as well as information on the current status of existing data systems, the cross-site evaluation will supply critical information to the field that will ultimately lead to rigorous collection and interpretation of the long-term outcomes of suicide prevention efforts.
To date, there have been few systematic studies of these mediating variables. Without the results of an evaluation, the interpretation of suicidal behavior outcomes (whether positive or negative) will remain impossible. For example, the causal chain upon which early identification gatekeeper training activities is based includes the early identification of at-risk youth, their referral to service, subsequent connection with those services, receipt of services, and the amelioration of at-risk circumstances ultimately resulting in the reduction of suicidal attempts and related deaths. In this scenario, one must first understand the impact of the gatekeeper training on referrals to service and subsequent connection to services; without positive outcomes in these intermediate areas, it is unrealistic to expect a positive impact on the ultimate outcomes associated with suicidal behavior.
The cross-site evaluation is the first comprehensive and systematic evaluation of the crucial mediating (proximal) outcomes of suicide prevention efforts such as awareness, knowledge, referrals, and service access. Currently, data collection for the cross-site evaluation is approved under Office of Management and Budget clearance (OMB No. 0930-0286), valid until August 2013. SAMHSA is requesting approval for revisions to the previously approved evaluation package.
The cross-site evaluation has four stages of information gathering that target the funded program activity areas: (1) Context Stage, (2) Product Stage, (3) Process Stage, and (4) Impact Stage1. Data collection activities have been tailored to the individual programmatic activities because programmatic approaches funded in the State/Tribal sites differ from those at the Campus sites. In addition to assessing the effectiveness of the GLS Suicide Prevention Program, information collected through the cross-site evaluation will continue to be used to report on SAMHSA’s National Outcome Measures (NOMs) that are relevant to program activities. Collected information from the evaluation will also be used to report on the Government Performance Reporting Act (GPRA) measures that are identified for this program.
Table 1 below summarizes the data collection instruments and their relation to Program Expectations listed in the Request for Applications (RFA) to which grantees applied
Table 1. Summary of Data Collection Activities
Type of Grantee |
Data Collection Instrument |
Purpose |
|
State/Tribal Grantees |
|
This instrument crosswalks with RFA requirement 2.0 that State/Tribal grantees implement outreach and other strategies to increase participation in, and access to, treatment or prevention services to underserved populations, and RFA requirement 2.1 that grantees fund services and practices that have a demonstrated evidence base and that are appropriate for the population(s) of focus. |
|
|
This instrument crosswalks with RFA requirement 2.0 that State/Tribal grantees ensure that educators, foster care, juvenile justice, childcare professionals, health, mental health and substance abuse professionals, and community care providers are properly trained to effectively identify youth who are at risk for suicide. |
||
|
This instrument crosswalks with RFA requirement 2.0 that State/Tribal grantees ensure that educators, foster care, juvenile justice, childcare professionals, health, mental health and substance abuse professionals, and community care providers are properly trained to effectively identify youth who are at risk for suicide. |
||
|
This instrument crosswalks with RFA requirement 2.0 that State/Tribal grantees ensure that educators, foster care, juvenile justice, childcare professionals, health, mental health and substance abuse professionals, and community care providers are properly trained to effectively identify youth who are at risk for suicide. |
||
|
This instrument crosswalks with RFA requirement 2.0 that State/Tribal grantees demonstrate collaboration among early intervention and prevention services or certify that entities will engage in future collaboration. |
||
|
This instrument crosswalks with RFA requirement 2.0 that grantees must form or participate in an existing public/private coalition of youth-serving institutions and agencies. |
||
|
This instrument crosswalks with RFA requirement 2.0 that grantees must form or participate in an existing public/private coalition of youth-serving institutions and agencies. |
||
|
This instrument crosswalks with RFA requirement 2.0 that State/Tribal grantees must provide timely referrals for appropriate community mental health care and treatment to youth who are at risk for suicide, and RFA requirement 2.2 that grantees report on the number of individual referred to and receiving mental health or related services. |
||
|
This instrument crosswalks with RFA requirement 2.2 that State/Tribal grantees report on the number of individuals screened for mental health or related interventions. |
||
|
This instrument crosswalks with RFA requirement 2.2 that State/Tribal grantees report on the number of individuals who have received training in prevention or mental health promotion. |
||
Type of Grantee |
Data Collection Instrument |
Purpose |
|
Campus Grantees |
|
This instrument crosswalks with RFA requirement 2.2 that Campus grant funds be primarily used to support infrastructure development including developing training programs for students and campus personnel, creating a networking infrastructure to link the institution with health care providers from the broader community, developing and implementing educational seminars, creating hotlines or promoting linkages to the National Suicide Prevention Lifeline, distributing informational materials that address signs of suicide, and distributing educational materials for families of students to increase awareness of potential behavioral health issues of students. |
|
|
This instrument crosswalks with RFA requirement 2.2.1 that Campus grantees develop training programs for students and campus personnel to respond effectively to students with mental and behavioral health problems. |
||
|
This instrument crosswalks with RFA requirement 2.2.1 that Campus grantees develop training programs for students and campus personnel to respond effectively to students with mental and behavioral health problems. |
||
|
This instrument crosswalks with RFA requirement 2.3 that Campus grantees report on the number of individuals exposed to mental health awareness messages. |
||
|
This instrument crosswalks with RFA requirement 2.3 that Campus grantees report on the number of individuals exposed to mental health awareness messages. |
||
|
This instrument crosswalks with RFA requirement 2.2.3 that Campus grantees develop and implement educational seminars that include promotion of help seeking and stigma reduction. |
||
|
This instrument crosswalks with RFA requirement 2.2.2 that Campus grantees that do not have comprehensive, campus-based mental health and behavioral health services, create a networking infrastructure to link the institution with health care providers from the broader community who can treat mental and behavioral health problems. |
||
|
This instrument crosswalks with RFA requirement 2.3 that Campus grantees report on the number of individuals in the mental health and related workforce trained in mental health-related practices/activities and the number of individuals who have received training in prevention or mental health promotion. |
||
Youth suicide is an enormous public health problem that takes the lives of many young persons; more than 4,000 adolescents and young adults die by suicide every year (American Association of Suicidology [AAS], 2010). Left in the aftermath are family members and friends who feel profound grief, guilt and shame at the loss of a young life. Although adolescent males, in comparison with adolescent females, die more frequently from suicide, adolescent females are more likely than adolescent males to attempt suicide (NAHIC, 2011). Of all youth populations, American Indian/Alaska Native males have the highest suicide rates (NAHIC, 2011). These prevalence data are likely an undercount of suicide deaths because of the manner in which cause of death is recorded on death certificates and because of the ambiguity of homicides and accidental deaths in which the person attempting suicide intentionally places himself or herself in harm’s way (U.S. Public Health Service, 1999).
Youth suicide can be linked to a number of mental health disorders as well as substance abuse. In 2003, the President’s New Freedom Commission on Mental Health recognized youth suicide prevention as a major priority. This was due to high rates of youth suicide that included large numbers of individuals who had been diagnosed with a mental illness and/or substance abuse disorder (Institute of Medicine, 2002). Adolescence is a time of rapid maturity and increasing responsibility, but many youth may experience a feeling of hopelessness about the future. This can apply particularly to college students and young adults between the ages of 20 and 24, the ages in which the highest youth suicide rates are observed (AAS, 2011). In a study by the American College Health Association (Reference Group Executive Summary, Spring 2012), 47% of college students reported feeling hopeless, 32% reported feeling so depressed they could barely function, and 8 % reported feeling suicidal.
Despite these high prevalence rates, up until 2005 with the initiation of the GLSMA, youth suicide remained a public health problem that largely went unaddressed. This is unfortunate because suicide is preventable. The majority of teens who attempt suicide display warning signs which if acted upon could prevent attempts. These may include indirect or direct suicide threats, an obsession with death, or giving away belongings. Also, because of the negative social norms that surround mental illness and suicide, youth often do not disclose their underlying emotional state or behavioral intentions. Consequently, it is extremely important to recognize these signs when exhibited, because the inability to do so may represent a missed opportunity for suicide prevention and intervention.
Suicide warning signs are less likely to occur, however, if protective factors are first recognized and taken into consideration. Youth who exhibit risk factors, such as depression, impulsivity, alcohol and substance abuse, and a history of trauma or abuse, are believed to have a greater potential for suicidal behavior (Cash, 2009). Examples of protective factors include problem-solving skills, effective clinical care, strong connections to family and community support, and restricted access to lethal methods for attempting suicide. Research into reducing the occurrence and subsequent burden of youth suicide has generated goals and strategies that build on the foundation of reducing risk factors while increasing protective factors (O’Connor, 2011).
However, suicide does not occur simply because of an inadequate blending of these factors nor will a universal solution result because of a proper combination of specific risk and protective factors. As emphasized in the following reports, it will take involvement from mental health, substance abuse, juvenile justice, primary care, education, the media, and other youth-serving organizations to successfully prevent the occurrence of youth suicide. Three documents, Reducing Suicide: A National Imperative (Institute of Medicine, 2002), The Surgeon General’s Call to Action to Prevent Suicide (U.S. Department of Health and Human Services [DHHS], Public Health Service, 1999), and the National Strategy for Suicide Prevention: Goals and Objectives for Action (U.S. DHHS, Public Health Service, 2012), all provide overlapping recommendations for how this problem can be effectively addressed.
The Institute of Medicine’s Reducing Suicide: A National Imperative (2002) highlighted the prevalence of suicide attempts and suicidal behaviors and emphasized the need for research to understand how to prevent suicide, while highlighting the challenges associated with such research. The Surgeon General’s Call to Action to Prevent Suicide (U.S. Public Health Service, 1999) highlighted the need for increased public awareness of the problem of youth suicide, interventions to enhance treatments, services, and programs, as well as a methodology to advance the science of suicide prevention, better known as AIM: awareness, intervention, and methodology. AIM is the foundation for the 15 key recommendations highlighted in the Surgeon General’s report. As a result of the collaboration of the Federal Government, many private and public stakeholders, and family members of persons who committed suicide, the AIM framework became the catalyst for a more thorough and comprehensive strategy—the National Strategy for Suicide Prevention: Goals and Objectives for Action (U.S. Public Health Service, 2001), first launched in 2001, and updated in 2012.
On October 21, 2004, Congress passed the Garrett Lee Smith Memorial Act (GLSMA), which was signed into law by President Bush, to mobilize efforts to support suicide prevention and early intervention. This act authorized the use of $82 million over 3 years to support States, Tribal communities, and colleges and universities to develop and implement various suicide prevention initiatives. This act strongly builds on Reducing Suicide: The Surgeon General’s Call to Action (U.S. Public Health Service, 1999), and the National Strategy for Suicide Prevention (U.S. Public Health Service, 2001) in its directive to use the scientifically proven methodologies identified in each of these reports to target those youth and young adolescents who have historically generated the highest suicide rates. Products of this effort, which encapsulate recommendations from each of these reports, include the GLS State/Tribal Youth Suicide Prevention and Early Intervention Program as well as the GLS Campus Suicide Prevention and Early Intervention Program. Objectives of these two programs include: providing early intervention and assessment for youth at risk for mental or emotional disorders; conducting information and awareness campaigns to inform gatekeepers, family members, peers, and others about the risk factors associated with youth suicide; and training physicians, educators, and providers to identify youth who exhibit at-risk behavior for suicide. This legislation not only provides support for implementing these strategies but also directs these programs to evaluate the effectiveness of their targeted interventions at the local level, and requires a cross-site evaluation and report to Congress.
In 2004, the Center for Mental Health Services (CMHS) of the Substance Abuse and Mental Health Services Administration (SAMHSA) announced the award of 14 State/Tribal cooperative agreements and 21 Campus grants for the GLS Youth Suicide Prevention and Early Intervention Program. In FY 2006, Congress authorized an additional $27 million for States, Tribal communities and colleges across the country. In May 2006, SAMHSA announced the award of an additional 8 State/Tribal cooperative agreements followed by an additional 14 State/Tribal cooperative agreements and 34 Campus grant awards in September 2006. An additional 2 State/Tribal cooperative agreements were awarded in June 2007.
In September 2008, 30 State/Tribal sites (7 of which were cohort 1 continuation awards) and 16 campuses (7 of which were continuation awards) received funding, followed in September 2009 by 18 State/Tribal cooperative agreements and 22 more Campus grants. In September 2011, SAMHSA awarded 38 State/Tribal cooperative agreements and 21 Campus grants; most recently, SAMHSA awarded 23 State/Tribal cooperative agreements, and 39 Campus grants. In sum, the GLS Youth Suicide Prevention and Early Intervention Program has funded 147 State/Tribal cooperative agreements and 153 campus grants.
Section 520E (g) of the GLSMA mandates a cross-site evaluation to be conducted concerning the effectiveness of the activities carried out under the State/Tribal Youth Suicide Early Intervention and Prevention Program. The GLSMA specifies that a report to Congress must be submitted:
“to analyze the effectiveness and efficacy of the activities conducted with grants, collaborations and consultations under [Section 520E].”
In addition, Section 520-E-2 (f) of the GLSMA mandates a cross-site evaluation of the Campus Suicide Prevention Program. The GLSMA specifies that a report must be submitted to Congress to include:
“an evaluation of the grant program outcomes, including a summary of activities carried out with the grant and the results achieved through those activities.” [including] “recommendations on how to improve access to mental and behavioral health services at institutions of higher education, including efforts to reduce the incidence of suicide and substance abuse.”
The cross-site evaluation will serve as a primary mechanism through which the initiative will be understood, improved, and sustained. As described previously, there is a dire need for a better understanding of suicide prevention efforts—first and foremost on the intermediate outcomes of these efforts and then, ultimately, on suicidal behavior itself. Because this suicide prevention initiative is the first to be federally funded, the rigor and utility of the evaluation and its findings are particularly critical. The emphasis of the cross-site evaluation is to gather the needed intermediate outcome information and data system infrastructure information across grantees to ensure that in future years efforts can move strategically forward on scientific ground to assess the impact of funded efforts on suicidal behavior. As such, the GLS cross-site evaluation will collect and analyze comprehensive data that focus on the context within which these programs are implemented, the products and services that are developed and utilized, the process through which programmatic activities are implemented, and impacts associated with those activities.
A Government contractor (referred to as the cross-site evaluator throughout this document) will coordinate data collection for the cross-site evaluation and provide support for its local-level implementation. Each grantee is required by the cooperative agreement and grant to both conduct a self-evaluation and to participate in the cross-site evaluation. In this partnership, the cross-site evaluator provides training and technical assistance (TA) regarding data collection and research design for the cross-site evaluation. In addition, the cross-site evaluator directly collects data, receives data from grantee data collection efforts, monitors data quality, and provides feedback to grantees. The data collection procedures, while systematically applied across funded sites, are specific to the local programmatic activities and infrastructure that support those activities. The data gathered through the cross-site evaluation will continue to be utilized for both grantee-specific and national assessments of the program.
Previously Approved Clearance
Currently, data collection for the cross-site evaluation is operating under OMB clearance (OMB No. 0930-0286), valid until August 2013. What follows is a brief description of the evaluation design included in the previously approved OMB request.
The four-stage cross-site evaluation is designed to answer the following overarching questions:
What types of prevention/intervention programs, services and products are used with youth identified as being at risk for suicidal behavior?
What is the reach of program services, products, and strategies?
To what extent does collaboration and integration influence referral mechanisms and service use?
What is the impact of program services, products, and strategies on knowledge, process, and behavior?
The cross-site evaluation stages are described below.
Context Stage. The purpose of the Context Stage is to gain an understanding of each grantee’s program plans, such as its target population, target region, service delivery mechanisms, service delivery setting, types of program activities to be funded, evaluation activities, existing data sources and availability of data elements to support the cross-site evaluation. Collectively, the information learned through the Context Stage is used to support other components of the cross-site evaluation.
Product Stage. The purpose of the Product Stage is to describe the development and utilization of prevention strategies at each State/Tribal and Campus grantee site. These prevention strategies may include: public awareness campaigns; outreach and awareness events; gatekeeper trainings; life skills development activities for youth; policies and protocols for responding to youth at risk; means restriction strategies; screening programs; and enhanced services, including early intervention, family support, and postsuicide intervention services.
Process Stage. The Process Stage of the cross-site evaluation assesses progress on key activities related to implementation of each grantee’s suicide prevention plans. Since there are differences between the State/Tribal and Campus program approaches toward suicide prevention, the type of information collected differs by type of grantee. Given that training is a major component of most grantees’ suicide prevention programs, this stage is designed to collect information on the major characteristics of the trainings from both State/Tribal and Campus grantees, such as the type of training as well the roles and demographics of participants. For Campus grantees, information is collected on participants’ intended use and satisfaction with the training immediately following the training experience. For a sample of training participants from both State/Tribal and Campus grantees, qualitative interviews are conducted 3 months following the training in order to understand how participants have utilized and retained the knowledge, skills and/or techniques they learned. For State/Tribal grantees, data collected through the Process Stage is used to examine collaboration among different organizations/agencies involved in youth referral networks and how these networks change over time. For Campus grantees, this component examines the suicide prevention exposure, awareness, and knowledge of faculty/staff and students.
Impact Stage. The purpose of the Impact Stage is to assess the impact that suicide prevention programs have on youth who are at risk for suicide. Existing data sources are used to assess the impact of program activities at the State/Tribal grantee and the Campus grantee levels. To assess the impact of State/Tribal program activities, existing information on youth referred for services and service receipt as a result of early identification activities is analyzed. To assess the impact of Campus program activities, existing administrative data related to the number of students who are at risk for suicide, the school retention rate, the number who seek services, and the type of services received, including emergency service use, is analyzed to determine the impact of Campus program activities on the student and campus populations.
Clearance Request
SAMHSA is requesting approval for revisions to the previously approved cross-site evaluation package. The fundamental design of the cross-site evaluation remains unchanged. Drawing upon our experience from the past seven years of data collection for the cross-site evaluation and feedback from grantees, SAMHSA has made improvements to the cross-site evaluation data collection instruments to reduce response burden, maximize utility of data for all stakeholders and deepen our understanding and knowledge of particular areas in the suicide prevention field. Revisions to the cross-site evaluation are summarized in Section A2b.
What follows is a description of the major components of the cross-site evaluation and their associated data collection instruments, revisions from the previously approved package, the uses of the information collected through the cross-site evaluation and the importance of the cross-site evaluation in addressing National Outcome Measures (NOMs) and GPRA reporting.
Cross-Site Evaluation Design and Data Collection Instruments
The various components of the cross-site evaluation are described below. Since there are differences between the State/Tribal and Campus program approaches to suicide prevention, the type of information collected differs by the type of grantee.
Context Stage
The purpose of the Context Stage is to gain an understanding of grantees’ program plans, such as grantee’s target population, target region, service delivery mechanisms, service delivery setting, types of program activities to be funded, evaluation activities, existing data sources and availability of data elements to support the cross-site evaluation. The cross-site evaluation team will use existing grant applications to gather information on grantees’ programs and the contexts in which they are implemented. Since information gathering in this stage utilizes existing grantee applications and will be conducted by the cross-site evaluation team, there is no formal data collection instrument and associated response burden for the grantees. Collectively, the information learned through the Context Stage is used to inform other components of the cross-site evaluation.
Product Stage
The purpose of the Product Stage is to describe the development and utilization of prevention strategies at each State/Tribal and Campus grantee site. The Prevention Strategies Inventory (PSI) (see Attachments B.1 and B.2) will be administered to one representative from each of the State/Tribal and Campus grantees. This inventory asks grantees to describe the different types of prevention strategies that they have implemented, such as public awareness campaigns; outreach and awareness events; gatekeeper trainings; life skills development activities for youth; policies and protocols for responding to youth at risk; means restriction strategies; screening programs; and enhanced services, including early intervention, family support, and postsuicide intervention services. There are two slightly different versions of the inventory for Campus grantees and State/Tribal grantees. Grantees will first complete the Baseline version. Thereafter, they will complete the Follow-up version on a quarterly basis over the duration of their grant period.
Process Stage
The Process Stage of the cross-site evaluation assesses progress on key activities related to implementation of each grantee’s suicide prevention plans. Since there are differences between State/Tribal and Campus program approaches to suicide prevention, the type of information collected differs by type of grantee. This stage includes several data collection instruments and extracting data from several different sources.
Training to enhance awareness, knowledge, early identification, and referral of youth at risk for suicide is a primary program activity for most State/Tribal and Campus grantees. Both Campus and State/Tribal grantees are required to report aggregate training participant information for all trainings conducted as part of their suicide prevention programs. These data are aggregated from existing data sources, some of which are attendance sheets, management information systems, etc. Grantees are responsible for aggregating these data and submitting them to the cross-site evaluation team in the format of the Training Activity Summary Page (TASP) (see Attachments H.3 and H.4). There are two slightly different versions of the TASP for Campus grantees and State/Tribal grantees.
To assess the content of the training, the participants’ intended use of the skills, knowledge learned, and satisfaction with the training experience, Campus grantees will administer the Training Exit Survey (TES) Individual Form (see Attachments C.1–C.4) to all participants immediately following the training. The Training Exit Survey (TES) Individual Form has two parts. While the core section of the survey will collect information on participant role, demographics, and satisfaction with the training experience, the modules tailored to particular training types will ask questions about participant knowledge, self-efficacy, and intent to use.
For State/Tribal and Campus grantees, the quantitative Training Utilization and Preservation Survey (TUP-S): State/Tribal and Campus Versions (see Attachments D.1 and D.4) will be administered to a random sample of trainees three months following the training in order to expand our knowledge on the utilization and retention of participants’ knowledge, skills and/or techniques learned through the training. The TUP-S will systematically measure gatekeeper behaviors and will include measures of self-efficacy, awareness and education efforts, and, most importantly, suicide identification behavior. The TUP-S will collect demographic information about individuals identified at risk, information about the subsequent referrals and/or supports provided by the trainee, and any available information about services accessed by the at-risk individual. There are two slightly different versions of the survey for Campus grantees and State/Tribal grantees. There is also a follow-up version of the form for State/Tribal grantees: the Training Utilization and Preservation Survey (TUP-S): State/Tribal Version 6-Month Follow-up (see Attachment D.9). This form will be administered 6 months after the completion of suicide prevention training.
In order to capture TUP-S data from the large number of State/Tribal trainees under the age of 18, the Training Utilization and Preservation Survey (TUP-S)-Adolescent Version (see Attachment D.7) will be piloted for 1 year. The survey will be administered to adolescents ages 12–18 years at 3 months after their participation in a prevention training offered by State/Tribal grantees. Two methods of administration will be piloted: one using a Web survey, and another using text messaging.
For Campus grantees, the Short Message Service Survey (SMSS) (see Attachment K) will be administered to a random sample of students, once in the first year of the grant and again in the third year. The four-question text message survey will assess student exposure to and participation in suicide prevention activities on campus, and will collect information on suicidal thoughts.
For a select group of up to four Campus grantees, the quantitative Student Awareness Intercept Survey (SAIS) (see Attachments J.1 and J.2) will be administered to approximately 400 students at the four participating campuses. Campuses implementing targeted suicide prevention campaigns will be identified and selected by reviewing grant applications and through technical assistance activities. The SAIS will collect information about exposure to suicide prevention outreach and awareness initiatives with targeted student populations; awareness of appropriate crisis interventions, supports, services, and resources for mental health seeking; knowledge of myths and facts related to suicide and suicide prevention; and attitudes toward mental health seeking, access and utilization of mental health services on campus. A follow-up version of the survey will be administered 3 months after baseline.
The Life Skills Activities Follow-up Interview (LAFI) (see Attachment I.1) will be administered to randomly selected participants of selected Campus trainings. This qualitative interview will address how students apply the skills and information learned through campus life skills and wellness activities aimed at enhancing protective factors. The instrument will be administered to up to 7 trainees from up to 5 selected campus trainings per year, for a total of up to 35 respondents per year.
For State/Tribal grantees, the Referral Network Survey (RNS) (see Attachment E) will be administered to representatives of youth-serving organizations and/or agencies that form referral networks supporting youth identified at risk. The RNS examines how collaboration and integration are used for sharing and transferring knowledge, resources, and technology among State/Tribal Program agencies and organizational stakeholders, how these networks influence referral mechanisms and service availability, policies and protocols regarding follow-up for youth who have attempted suicide and are at risk for suicide, as well as access to electronic databases. The RNS will be administered to referral networks twice, once in the first year of the grant and again in the third year. State/Tribal grantees will also be asked to complete a brief survey, the Coalition Profile (CP) (see Attachment F), about their primary coalition’s mission and structure. The Coalition Survey (CS) (see Attachment G), which measures an organization’s involvement in the grantee’s suicide prevention coalition, will then in years 1 and 3 of the grant be administered to up to 10 coalition members from each State/Tribal grantee’s primary coalition.
Impact Stage
The purpose of the Impact Stage is to assess the impact that the suicide prevention programs have on youth who are at risk for suicide. Existing data sources are used to assess the impact of program activities at the State/Tribal grantee and the Campus grantee levels.
To assess the impact of State/Tribal program activities, existing information on youth referred for services and service receipt as a result of early identification activities is analyzed. The Early Identification, Referral and Follow-up Analysis (EIRF) (see Attachment H.1) require State/Tribal grantees to share existing data with the cross-site evaluation team on the number of youth identified as being at risk as a result of early identification activities, referred for services, and who presented for services. The type of information that will be shared with the cross-site evaluation includes basic demographic information; types of service referrals; and types of services received, which includes mental health assessments, mental health treatment, emergency services, and nontraditional support services.
State/Tribal grantees are also required to report aggregate screening information for all youth screened as part of their suicide prevention programs. These data are aggregated from existing data sources. Grantees are responsible for aggregating these data and submitting them to the cross-site evaluation team in the format of the Early Identification, Referral and Follow-up Screening Form (EIRF-S) (see Attachment H.2).
To assess the impact of Campus program activities, the cross-site evaluation team will request campus sites to engage in a MIS Data Collection Activity (see Attachments H.5 and H.6) to submit existing administrative data related to the number of students who are at risk for suicide, the school retention rate, the number of attempted or completed suicides among students who live on and off campus, the number who seek services, and the type of services received, including emergency services, is analyzed to determine the impact of Campus program activities on the student and campus populations.
Revisions
Below is a summary of revisions to the previously approved cross-site evaluation package and the rationale behind each of the changes:
Through the most recent OMB clearance for the cross-site evaluation was requested and approved for 3 years of data collection until August 2013. Respondent burden for the revised clearance is calculated for the next 3 years of data collection from August 2013 to August 2016.
The number of grantees for which burden is calculated is 121 (61 State/Tribal grantees and 60 Campus grantees), which represents the number of currently active grantees. It should be noted that SAMHSA is using this number as an estimate of the number of grantees that are active per year. Fifty-nine grantees (out of the 121 grantees) were funded in September 2011 and will reach the end of their grant period in September 2014. At that point, additional grantees may be funded. Therefore, SAMHSA is estimating that in a given year, there would be 121 active grantees.
For the Product Stage, the previously approved Prevention Strategies Inventory (PSI) (see Attachments B.1 and B.2) has improved response options that better capture subpopulations targeted for prevention strategies. Response options now include the following: American Indian/Alaska Native; Survivors of Suicide; Individuals who engage in nonsuicidal self-injury; Suicide attempters; Individuals with mental and/or substance abuse disorders; Lesbian, gay, bisexual, and transgender populations; Veterans, active military, or military families; Hispanic or Latino population. Additional guidance has also been provided for categorizing prevention strategies that fit in multiple categories. These changes enhance the utility and accuracy of the data collected.
In the Process Stage, several improvements have been made to instruments designed to collect data on gatekeeper training. In order to collect information about the settings of trainings and the training goal, as well as the follow-up plans of grantees, brief questions have been added to the previously approved Training Exit Survey (TES) Cover Page. The form has also been renamed the Training Activity Summary Page (TASP) (see Attachments H.3 and H.4). The various modules of the previously approved Training Exit Survey (TES) Individual Form (see Attachment C.1–C.4) that collect information on trainee knowledge and self-efficacy will continue to be collected by Campus grantees, but will no longer be collected by States and Tribes.
The previously approved Training Utilization and Preservation Survey (TUP-S) (see Attachment D.1) will continue to be administered to States/Tribes, but an additional 6-month follow-up version of the form—the Training Utilization and Preservation Survey (TUP-S) 6-Month Follow-Up (see Attachment D.9)—will also now be administered to States/Tribes.
Two new versions of the TUP-S will also be administered: one to Campuses (the Training Utilization and Preservation Survey-Campus Version (TUP-S-C)) and one to adolescents under the age of 18 (the Training Utilization and Preservation Survey-Adolescent Version) (see Attachments D.4 and D.7). The campus version allows for the collection of information about the utilization and retention of participants’ knowledge, skills and/or techniques learned through trainings conducted on campuses—information that, up until this point, has been collected only by States and Tribes. It will be piloted for one year. The adolescent version of the survey increases the comprehensiveness of the evaluation, as it allows for the collection of training utilization and retention data among adolescents under the age of 18, who represent more than a fifth of the trainees from States and Tribes, but who heretofore have not participated in the TUP-S. The Campus version of the survey will be implemented in the same way as the State/Tribal version: a random sample of trainees will be contacted 3 months following the training to participate in a quantitative telephone survey. The 1-year pilot of the Adolescent version of the survey will be implemented with grantees sponsoring trainings for youth as part of their grant program. Two methods to reach adolescents to complete the TUP-S will be piloted: one using a Web survey, and another using a SMS, or text message, survey. A module of questions to assess resiliency has also been added to the previously approved TUP-S State/Tribe version.
A new instrument, the Life Skills Activities Follow-up Interview (LAFI) (see Attachment I.1) will be added to the Process Stage. Campuses devote about a tenth of their GLS resources to support life skills and wellness activities aimed at enhancing protective factors among participating students, yet there is little information about how these skills and knowledge are utilized. The LAFI, a qualitative interview with participants of selected Campus trainings, will help address this gap.
The Referral Network Survey (RNS) (see Attachment E) utilized in the Process Stage has undergone several changes. It has been revised to gather more detail about the type, level, and quality of collaboration between agencies, including barriers, facilitators, and outcomes of the collaboration. The mode of administration for this survey will be changed from phone to the Web to boost response rates. Also, as part of the Process Stage, the Coalition Survey (CS) (see Attachment G) will be administered to States and Tribes. The CS complements the RNS in that it collects information directly from coalition members about collaboration efforts. Prior to the administration of the Coalition Survey, State/Tribal grantees will participate in a brief survey, the Coalition Profile (CP) (see Attachment F), to gather information about each grantee’s primary coalition, such as its mission and structure, without having to ask each of the coalition members these questions during the Coalition Survey.
The previously approved Suicide Prevention, Exposure and Awareness Knowledge Survey for Students (SPEAKS-S) that forms part of the Process Stage will be replaced by the Short Message Service Survey (SMSS) (see Attachment K) in all campuses, and complemented by the Student Awareness Intercept Survey (SAIS) (see Attachments J.1 and J.2) in up to four selected campuses. By using technology relevant to the college-aged population, the SMSS aims to increase response rates, as well as measure student’s knowledge of suicide prevention—a SAMHSA GPRA indicator. Similarly, the SAIS collects important information on students’ exposure to suicide prevention outreach and awareness initiatives in targeted populations at up to four selected campuses. Campuses will be ranked on their level of emphasis on awareness campaigns based on a review of grantee applications and through technical assistance activities, and the four highest ranking campuses will be selected for participation in the SAIS.
The previously approved Early Identification, Referral and Follow-up Aggregate Screening Form (EIRF-S) (see Attachment H.2) that is part of the Impact Stage has been modified to collect the geographical location of screening events. The form has also been renamed the Early Identification, Referral and Follow-up Screening Form. The previously approved Early Identification, Referral and Follow-Up Analysis form (see Attachment H.1) has also been modified to collect the geographical location of the setting in which the youth was identified, and the setting in which the youth received services in an effort to track service availability and accessibility. On both forms, questions and response options have been modified for clarity.
The form used to capture data from the MIS Data Collection Activity (see Attachments H.5 and H.6), part of the Impact Stage, has been modified to allow grantees to capture data on the number of attempted or completed suicides among students who live on and off campus.
The previously approved Campus Infrastructure Interviews (CIFI) that were part of the Process Stage have been removed in an effort to reduce burden.
Three previously approved instruments collected by a subset of Campus grantees have also been removed in an effort to reduce burden.
The previously approved Training Utilization and Preservation Interview (TUP-I) that was part of the Process Stage has been removed, as most of the information collected in this interview is also captured in the Training Utilization and Preservation Survey (TUP-S).
Uses of Information Collected through the Cross-Site Evaluation
The GLS cross-site evaluation is also in alignment with and provides data sources to track SAMHSA’s Strategic Initiatives, which the agency will use to guide its work through at least 2014. The eight strategic initiatives are designed to focus SAMHSA’s work on improving lives and capitalizing on emerging opportunities. In particular, the GLS cross-site evaluation responds to the following strategic initiatives:
Prevention of Substance Abuse and Mental Illness: The promotion of positive mental health and the prevention of substance abuse and mental illness have been key parts of SAMHSA’s mission to reduce the impact of substance abuse and mental illness on America’s communities. This strategic initiative includes four goals with embedded objectives and action steps. Of those, the GLS program and data collection associated with the cross-site evaluation contribute to the following:
Goal 1.3: Prevent suicides and attempted suicides among populations at high risk, especially military families, LGBTQ youth, American Indians, and Alaska Natives.
Without the information gathered from the cross-site evaluation, SAMHSA will not have a systematic way of assessing the degree to which State/Tribal grantees are able to link at-risk youth to services. Data collected through the cross-site evaluation’s Early Identification Referral and Follow-up Analysis (EIRF) provide a critical step in monitoring and understanding service linkages that promote life-saving and prevent suicide, especially among high-risk populations such as LGBTQ youth, American Indians, and Alaska Natives.
Likewise, little information is available about the extent to which gatekeeper training actually supports suicide prevention and intervention with high-risk youth. The cross-site evaluation provides important information on trainee perceptions of their ability to recognize and appropriately respond to suicide risk factors as a result of training activities. Similarly, cross-site evaluation data describing how the training they received increased referrals for mental health services and/or social support will add to the existing knowledge base about the degree to which suicide prevention programs promote life-saving.
The cross-site evaluation also measures the success of suicide prevention programs among another population: college students. Existing research shows that college students face enormous pressures and often have difficulties dealing with these stressors (as cited in the GLSMA, Public Law 108-355). Yet little is known about whether suicide prevention activities are reaching the students who are being targeted to receive services. The cross-site evaluation addresses this knowledge gap. By analyzing existing data from Campus grantees’ management information systems, the cross-site evaluation also assesses the extent to which suicidal behavior, including suicide attempts and completions, has decreased as a result of the GLS program.
Military Families: SAMHSA is taking the lead on filling the gaps that exist in the care available in communities for military families. Increased coordination is needed between military health care systems and the behavioral health care system. This strategic initiative includes four goals with imbedded objectives and action steps. Of those, the GLS program and data collection associated with the cross-site evaluation contribute to the following:
Goal 3.3: Promote the behavioral health of military families with programs and evidence-based practices that support their resilience and emotional health and prevent suicide.
Data from the cross-site evaluation provides SAMHSA a measure of the degree to which GLS resources are distributed and used in such a way that they reach the highest need populations—those most affected by and at risk for suicide—including military families. Through the cross-site evaluation, SAMHSA is able to identify gaps in service provision. The Prevention Strategies Inventory (PSI) provides critical information to SAMHSA about the extent to which evidence-based suicide prevention trainings are implemented with high-risk populations, including military personnel.
Data, Outcomes and Quality Initiative: SAMHSA has highlighted the importance of supporting programming decisions with high quality data and of transparency in these decisions by making data readily available to the public. This initiative includes four goals with imbedded objectives and action steps. Of those, the GLS cross-site evaluation is guided by the following:
Goal 7.3: Improving the quality of SAMHSA’s program evaluations and services research.
Goal 7.4: Improving the quality and accessibility of surveillance, outcome and performance, and evaluation information for staff, stakeholders, funders, and policymakers.
The stage-specific utility and contribution of the cross-site data collection to SAMHSA’s mission and decision making are described below:
Context Stage. Specifically, the cross-site evaluation team and SAMHSA will use information collected through the Context Stage to assess the availability of existing data sources to report on program activities and to support GPRA reporting. Assessing the availability of existing data will also support analyses conducted as part of the Impact Stage of the cross-site evaluation.
Product Stage. Specifically, SAMHSA will use information gained through the cross-site evaluation to describe the prevention strategies that were developed and/or utilized as part of suicide prevention programs. Information collected as part of the Product Stage will inform other States and Tribal communities, as well as campuses, across the country as to what products and services support suicide prevention.
Process Stage. As part of the Process Stage, specific findings related to training activities will inform SAMHSA, States, Tribal communities and Campuses on what type of training activities are being implemented via these funded suicide prevention programs, who is being training, the intended and actual utilization and impact of those trainings, and the overall satisfaction with training experiences. This information will assist grantees in implementing training activities as part of their suicide prevention program. In addition, information collected as part of the training exit survey will continue to inform grantees about any necessary training modifications and/or enhancements; and follow-up training information will help inform the extent to which training activities are having an impact on youth in the community. Also, as part of the Process Stage, specific findings related to referral networks will inform SAMHSA and State/Tribal suicide prevention efforts across the country by describing the organizations involved in referral networks, what types of relationships exist, the extent to which grant funding enhanced the development of referral networks, and to what extent these networks are being used to support high-risk youth. For funded State/Tribal grantees, information collected during the first administration of the State/Tribal referral network survey will assist State/Tribal grantees in further developing their referral networks in years 2 and 3 of grant funding.
Impact Stage. SAMHSA will use information gained through the Impact Stage to measure the extent to which grant funding is used to connect at-risk youth to services and to promote life-saving. Specifically, the Impact Stage measures the extent to which State/Tribal grantees are able to link at-risk youth to services; assesses the timeliness of service provision; assesses the number of at-risk youth on campuses who seek services, the type of services provided and the number of suicide attempts and completions on campus.
In totality, the data collected as part of the cross-site evaluation will continue to be useful to SAMHSA and its partners, other Federal agencies and administrators, the State/Tribal grantees, the Campus grantees, legislators, the field of suicide prevention, individual youth and their families, and the communities in which they live. Comprehensive information gathered from multiple sites at various levels and stages of programmatic activity will continue to augment the existing knowledge base tremendously.
In addition, and of equal importance, SAMHSA will continue to use the results from the cross-site evaluation to develop policies and provide information to other States/Tribal communities, and campuses regarding the development and implementation of suicide prevention programs, as well as develop and refine future funding priorities of the GLS Suicide Prevention Program or similar programs.
Finally, information from the cross-site evaluation helps other SAMHSA programs in developing and implementing suicide prevention activities, designing comprehensive data collection efforts to monitor those activities, and reporting to local and Federal stakeholders. If these data are not collected, policymakers and program planners at the Federal and local levels will not have the necessary information to determine the extent to which suicide prevention activities are effective and having an impact on youth at risk for suicide. Without this evaluation, Federal and local officials will not know whether the suicide prevention programs implemented as part of the GLSMA had an impact on suicide prevention, the identification of at-risk youth, and whether GLS grantee programs are meeting the goals of the GLSMA. SAMHSA will also use data collected as part of the cross-site evaluation to provide objective measures of its progress toward meeting targets of key performance indicators put forward in its annual performance plans as required by law under the GPRA.
Addressing National Outcome Measures (NOMs) and GPRA Reporting
The cross-site evaluation was designed in part to support SAMHSA’s performance measurement and management efforts. In assessing the effectiveness of each State/Tribal and Campus suicide prevention program, the cross-site evaluation will evaluate the GLS Suicide Prevention and Early Intervention Program as a whole. This is a critical step toward assessing the ability of the program to achieve many of the goals implied by GPRA indicators and SAMHSA National Outcome Measures (NOMs). The cross-site evaluation design reflects the intention of SAMHSA to implement performance management and accountability in all programs.
The cross-site evaluation design addresses the three-tiered SAMHSA NOMs and GPRA measurement approach by incorporating relevant client-level, training-related and infrastructure development outcome measures. The SAMHSA client-level NOM domains to date have been developed to address outcomes related to mental health and substance abuse treatment programs and substance abuse prevention programs. Because the GLS Suicide Prevention Program focuses on suicide and prevention, rather than treatment and/or substance abuse, not all client-level measures included in the existing 10-domain client-level NOM framework are appropriate for suicide prevention. To further explain lack of appropriateness, the majority of funding across both State/Tribal and Campus programs is dedicated to the early identification and referral of youth at risk for suicide, and enhancing awareness about suicide. Currently, no funds are devoted to the provision of treatment. As a result, data collection activities and resources, as well as monitoring of program focus, should be appropriately placed on the activities being funded and related outcomes. Furthermore, while many of the treatment NOM domains are considered potential distal outcomes for those youth or university/college students who are identified as at risk, referred into service, and receive treatment (e.g., decreased mental health symptomatology, abstinence from drug and alcohol use), the reporting of this type of information requires, among other things: (1) the receipt of mental health treatment which the GLS Suicide Prevention funds are not currently supporting, (2) the tracking of individuals to request self-reported information which the GLS suicide prevention grantees are not resourced to accomplish, and/or (3) the access to existing treatment MISs that the GLS suicide prevention grantees typically do not access, given their strategic plans and partnership structure.
To that end, client-level measures that are viable for GLS suicide prevention program activities have been abstracted from the existing 10-domain structure, and appropriate Infrastructure, Prevention and Promotion (IPP) Indicators have been proposed. Jointly reporting on these indicators will provide a comprehensive performance measurement and management approach that will represent the breadth of GLS program activities and their reach. A summary of the client-, training-, and infrastructure-level indicators that will be used to facilitate NOMs/GPRA reporting for the GLS Suicide Prevention Program is described below and in Table 2.
Client-level NOMs: As detailed above, several of the client-level NOM domains are considered inappropriate for the GLS Suicide Prevention and Early Identification Program. Specifically, domains related to decreased symptomatology, increased stability in housing, decrease in juvenile justice involvement, retention in substance use treatment, and abstinence from alcohol are considered unviable for the reasons described in the previous section. Several client-level domains and IPP Indicators, however, are relevant for GLS suicide prevention programs because they specify outcomes related to early identification and referral of youth – specifically, access to mental health services, increased social supports, use of evidence-based programs/practices, and retention in education for university/college students. Early identification activities are a key component of GLS suicide prevention programs and focus on the use of evidence-based practices/approaches (IPP: use of evidence-based practice) to identify youth or university/college students at risk for suicide and connecting those individuals to appropriate mental health or emergency services (IPP: access to service) and support services (NOM: social supports and connectedness). In addition, because Campus suicide prevention activities are being implemented with university/college students, the NOM related to education retention will be reported for the Campus program. Data from the cross-site evaluation will be used to facilitate reporting on these client-level NOMs and IPP indicators.
Training-related Proposed Domains: Because the GLS Suicide Prevention Program focuses on prevention rather than treatment, a large amount of grant funds, particularly in the State/Tribal sites, are being dedicated to gatekeeper training and early identification activities. Appropriate training-level measures become critically important for the consistent performance measurement and management for the GLS State/Tribal and Campus programs. Specifically, access to training, satisfaction with training experience, increased knowledge as a result of training, and intended use of the acquired skills are incorporated into the cross-site evaluation design of the State/Tribal program activities.
Infrastructure Proposed Domains: Across the GLS Suicide Prevention Programs (i.e., State/Tribal and Campus programs), the prevention activities are being collectively implemented in an effort to build and strengthen suicide prevention infrastructures at the State level and the Campus level. These activities include public information campaigns, education campaigns, gatekeeper trainings, product development, and coalition building. In an effort to facilitate consistent performance measurement and management of infrastructure development and change, the National Strategy for Suicide Prevention objectives has been used as a framework for selecting relevant infrastructure indicators. Specifically, the proposed infrastructure domains are: promoting awareness, the provision and implementation of suicide prevention activities across sectors, and improving and expanding suicide attempt and completion surveillance.
Table 3 provides a cross-walk of the proposed GPRA indicators for the GLS Suicide Prevention Program and details the cross-site evaluation State/Tribal and Campus data source for each proposed indicator.
Table
2. SAMHSA National Outcome Measure Crosswalk with the Cross-site
Evaluation of the GLS Suicide Prevention and Early
Intervention
Program
Client-Level Outcomes |
|||
NOMS/LPP Domain |
NOMs Outcome |
Cross-Site Evaluation State/Tribal Data Source |
Cross-Site Evaluation Campus Data Source |
Access/ Capacity |
Increased Access to Services (Service Capacity) |
Information obtained through the Early Identification, Referral and Follow-up (EIRF) analysis will provide a measure of service accessibility for the State/Tribal suicide prevention programs and a measure of emergency service use. The EIRF process will identify the number of youths who are identified as at risk for suicide through program activities, the number who are referred for services, and the number who receive services and the type. This will provide a measure of service capacity among State/Tribal suicide prevention programs. |
In the context stage of the evaluation, the cross-site team will identify existing sources of information that can be obtained from campuses to facilitate the reporting of access to services and service capacity on campuses involved in early identification activities. The cross-site team will identify existing data elements of interest and request that campuses share those data with the cross-site evaluation for analyses. This will include a measure of emergency service use among campus student populations. |
Social Connectedness |
Increased Social Supports/Social Connectedness
|
Information obtained through the Early Identification, Referral and Follow-up (EIRF) analysis will identify the number of youths who are identified as at risk for suicide and who are referred for social supports. In addition, the Prevention Strategies Inventory (PSI) will collect information on life skills development activities and cultural activities that aim to strengthen youth’s sense of social connectedness. |
The Short Message Service Survey (SMSS) and the Student Awareness Intercept Survey (SAIS) inquire about students’ involvement and connectedness to the campus as well as their help-seeking behaviors. In addition, the Prevention Strategies Inventory (PSI) will collect information on life skills and wellness activities that increase students’ sense of connectedness to the campus community. The Life skills Awareness Follow-up Interview (LAFI) will collect information on how students have applied the knowledge learned through life skills and wellness trainings/activities. |
Table
2. SAMHSA National Outcome Measure Crosswalk with the Cross-site
Evaluation of the GLS Suicide Prevention and Early
Intervention
Program (continued)
Client-Level Outcomes |
|||
NOMs/LPP Domain |
NOMs Outcome |
Cross-Site Evaluation State/Tribal Data Source |
Cross-Site Evaluation Campus Data Source |
Use of Evidence-Based Practice |
Use of Evidence-based Practices |
The Prevention Strategies Inventory (PSI) documents on a quarterly basis the programs that have been implemented as part of the GLS suicide prevention program. The extent to which grantees use evidence-based programs can be analyzed by looking at whether the programs reported by grantees are part of the SPRC/AFSP Evidence-Based Practices Project and SAMHSA’s National Registry of Evidence-Based Programs and Practices. |
The Prevention Strategies Inventory (PSI) documents on a quarterly basis the programs that have been implemented as part of the GLS suicide prevention program. The extent to which grantees use evidence-based programs can be analyzed by looking at whether the programs reported by grantees are part of the SPRC/AFSP Evidence-Based Practices Project and SAMHSA’s National Registry of Evidence-Based Programs and Practices. |
Education Retention |
Student Retention Rate |
Not applicable: States/Tribal funds are not specifically targeting at-risk, school-based populations, but rather statewide youth in a variety of community and organizational settings. |
The Context Stage of the evaluation will identify the source of information for student retention. Campuses will be required to share aggregate student retention rates with the cross-site evaluation team. |
Table
2. SAMHSA National Outcome Measure Crosswalk with the Cross-site
Evaluation of the GLS Suicide Prevention and Early
Intervention
Program (continued)
Training Related Outcomes |
|||
Proposed Domain |
Proposed Outcome Measure (National Strategy for Suicide Prevention [NSSP] Goal) |
Cross-Site Evaluation State/Tribal Data Source |
Cross-Site Evaluation Campus Data Source |
Implement Training to Identify At-Risk Behavior |
Provide training on suicide prevention to: all community groups/providers, mental health and substance abuse services providers, and other health professionals (NSSP Goal 7: Objective 7.1, 7.2 and 7.3). |
To measure the number of community groups/providers, mental health and substance abuse services providers, and other health professionals who have received training as part of GLS-funded programs, the Training Activity Summary Page (TASP) will document the number trained and the role for each trainee. |
To measure the number of community groups/providers, mental health and substance abuse services providers, and other health professionals who have received training as part of GLS-funded programs, the Training Activity Summary Page (TASP) will document the number trained and the role for each trainee. |
Infrastructure Development Outcomes |
|||
Proposed Domain |
Proposed Outcome Measure (National Strategy For Suicide Prevention [NSSP] Goal) |
Cross-Site Evaluation State/Tribal Data Source |
Cross-Site Evaluation Campus Data Source |
Promote Awareness |
Develop, implement, and evaluate communication efforts designed to reach defined segments of the population and to increase knowledge of suicide prevention (NSSP Goal 2: Objective 2.1 and 2.4). |
To measure the implementation of public information campaigns in GLS-funded States/ Tribes, the Prevention Strategies Inventory (PSI) will document on a quarterly basis all public information products and services that were implemented as part of each grantee’s suicide prevention program. |
To measure the implementation of public information campaigns in GLS-funded Campuses, the Prevention Strategies Inventory (PSI) will document on a quarterly basis all public information products and services that were implemented as part of each grantee’s suicide prevention program. The Short Message Service Survey (SMSS) and the Student Awareness Intercept Survey (SAIS) will measure the reach and penetration of suicide prevention communication campaigns on GLS-funded Campuses. |
Table
2. SAMHSA National Outcome Measure Crosswalk with the Cross-site
Evaluation of the GLS Suicide Prevention and Early
Intervention
Program (continued)
Infrastructure Development Outcomes |
|||
Proposed Domain |
Proposed Outcome Measure (National Strategy For Suicide Prevention [NSSP] Goal) |
Cross-Site Evaluation State/Tribal Data Source |
Cross-Site Evaluation Campus Data Source |
Promote Awareness |
Increase communication efforts conducted online that promote positive messages and support safe crisis intervention strategies (NSSP Goal 2: Objective 2.3). |
To measure the extent that the Web is utilized to disseminate information on safe crisis intervention strategies, the Prevention Strategies Inventory (PSI) will document on a quarterly basis all public information efforts that involve Web site development or enhancements for the purposes of disseminating suicide prevention information. |
To measure the extent that the Web is utilized to disseminate information on safe crisis intervention strategies, the Prevention Strategies Inventory (PSI) will document on a quarterly basis all public information efforts that involve Web site development or enhancements for the purposes of disseminating suicide prevention information. |
Develop and Implement Prevention Programs |
Strengthen the coordination, implementation, and evaluation of comprehensive State/Tribal, and local suicide prevention programming (NSSP Goal 5: Objective 5.1). |
The Referral Network Survey (RNS) and the Coalition Survey (CS) will assess the extent to which State/Tribal grantees have implemented a coordinated network of referral supports and services for youth identified as at risk for suicide. The Early Identification, Referral and Follow-up Survey (EIRF) will measure the effectiveness of State/Tribal grantee referral networks in providing services to youth identified as at risk and in following-up with youth at risk to ensure they receive appropriate services. As part of the cross-site evaluation, an annual evaluation progress report will be provided by all grantees to document evaluation progress. |
As part of the cross-site evaluation, an annual evaluation progress report will be provided by all grantees to document evaluation progress. The Short Message Service Survey (SMSS) and the Student Awareness Intercept Survey (SAIS) will also measure the reach and penetration of suicide prevention communication campaigns on GLS-funded Campuses. |
Table
2. SAMHSA National Outcome Measure Crosswalk with the Cross-site
Evaluation of the GLS Suicide Prevention and Early
Intervention
Program (continued)
Infrastructure Development Outcomes |
|||
Proposed Domain |
Proposed Outcome Measure (National Strategy For Suicide Prevention [NSSP] Goal) |
Cross-Site Evaluation State/Tribal Data Source |
Cross-Site Evaluation Campus Data Source |
Develop and Implement Prevention Programs |
Encourage community-based settings to implement effective programs and provide education that promote wellness and prevent suicide related behaviors (NSSP Goal 5: Objective 5.2). |
To measure the extent to which evidence-based programs are being implemented in agencies and organizations serving families and youth, the Training Activity Summary Page (TASP) will document the evidence-based programs that are being implemented as part of GLS-funded programs, and in what capacity. This includes child welfare offices, family service offices, correction facilities, law enforcement, health care facilities, juvenile probation offices, community-based organizations, etc. The PSI also documents on a quarterly basis the programs that have been implemented as part of the GLS suicide prevention program, whether these programs are evidence-based, and in which setting these programs are implemented. |
To measure the extent that evidence-based programs are being implemented in agencies and organizations serving families and youth, the Training Activity Summary Page (TASP) will document the evidence-based programs that are being implemented as part of GLS-funded programs, and in what capacity. The PSI also documents on a quarterly basis the programs that have been implemented as part of the GLS suicide prevention program. The extent to which grantees use evidence-based programs can be analyzed by looking at whether the programs reported by grantees are part of the SPRC/AFSP Evidence-Based Practices Project and SAMHSA’s National Registry of Evidence-Based Programs and Practices. The Life Skills Activities Follow-up Interview (LAFI) will also document the extent to which training participants utilized the knowledge and skills learned in wellness activities aimed at increasing protective factors. |
Table
2. SAMHSA National Outcome Measure Crosswalk with the Cross-site
Evaluation of the GLS Suicide Prevention and Early
Intervention
Program (continued)
Infrastructure Development Outcomes |
|||
Proposed Domain |
Proposed Outcome Measure (National Strategy For Suicide Prevention [NSSP] Goal) |
Cross-Site Evaluation State/Tribal Data Source |
Cross-Site Evaluation Campus Data Source |
Improve and Expand Surveillance Systems |
Improve and expand State/Tribal, territorial, and local public health capacity to routinely collect, analyze, report, and use suicide-related data to implement prevention efforts and inform policy decisions (NSSP Goal 11: Objective 11.3). |
As part of the cross-site evaluation, an annual evaluation progress report will be provided by all grantees to document evaluation progress. Included in this process will be an assessment of whether GLS-related program data are integrated from multiple data management systems and whether these data are utilized in annual reports. |
As part of the cross-site evaluation, an annual evaluation progress report will be provided by all grantees to document evaluation progress. Included in this process will be an assessment of whether GLS-related program data are integrated from multiple data management systems and whether these data are utilized in annual reports. |
The GLS Suicide Prevention and Early Intervention Program evaluation approach, the process through which it was developed, and the training and TA that will be provided to grantees, have each fully intersected with utilization-focused Federal program accountability requirements (i.e., GPRA and NOMs). Therefore, a recommendation has been made that SAMHSA submit the cross-site evaluation package to the Office of Management and Budget.
Every effort was made to limit burden on individual respondents who participate in the cross-site evaluation through the use of technology. Data collection instruments will be administered via the Web and telephone. Below is a description of the Web-based data collection and management system and the CATI and SMSS technology that will be used for data collection.
Web-based data collection and management system
A Web-based data collection and management system will be used to facilitate data collection by program staff, program participants, key stakeholders, students, and Campus faculty/staff. The Web-based data collection and management system will serve two functions: (1) as a data entry tool for program staff and cross-site evaluation staff to enter cross-site evaluation information or data elements, and (2) as a data collection tool for administering Web-based surveys to respondents. All cross-site evaluation data obtained either through direct entry by program and/or evaluation staff or through Web-based surveys will be stored in the Web-based data collection and management system. The Web-based data collection and management system reduces evaluation burden for the grantees and allows ease of access to data for program personnel and cross-site evaluation team members.
The Web-based system is a completely secure system that maintains privacy through the provision of five different levels of password-protected access to site specific and aggregate data. All data collected will be stored in the central data repository that will allow for the analysis and summary of information within and across surveys. The five distinct user security levels include:
The Cross-site Administrator will have access to site-specific data from all grantee sites stored in the data collection and management system, and will have access to aggregate reports available on the system using this privilege level.
The Site Administrator will have access to site-specific data from the data collection and management system, and will have access to site-specific and aggregate reports available on the system. The administrator will also be able to view the number of instruments that have been completed and submitted. One individual per community will be designated the Site Administrator.
A Site User has the capability to access information available on the system, but will be restricted from accessing datasets.
The Contact User will have access to aggregate information available on the repository. The Contact User will not have rights to download datasets, nor to access information specific to a grant-funded community.
Data contributors are data collectors and survey respondents who will have the capability to enter data into the Web-based system, but will have no other privileges.
The cross-site evaluation team will provide training and TA to support grantees in implementing the cross-site evaluation and in using data at the site level. Program personnel will be trained to utilize the data collection and management system and will be provided with a user’s manual.
Only individuals (Cross-Site Administrator and Site Administrator) with security access at the site administrator level are allowed access to raw data. To protect potential misuse of that data, specifically related to the inadvertent identification of respondents as a function of their unique demographic/workforce characteristic profile, the cross-site evaluation team will restrict access to raw datasets to designated individual(s), and the Site Administrator of the SPDC will be asked to sign a data use agreement. Within the context of protecting from inadvertent identification, this agreement will stipulate who, how, and under what circumstances the raw data can be analyzed/reported. For example, the cross-site evaluation team will obtain an agreement from each Site Administrator agreeing not to report categories in which less than 10 cases exist and to stipulate who will have access to raw data. Further, the agreement will indicate that no attempt, through complex analysis and with outside information, will be made to ascertain from the datasets the identity of particular persons. Attachment L is the agreement that will be utilized.
A System of Records Notice (SORN) and HHS Privacy Impact Assessment (PIA) form have been completed for the Personally Identifiable Information (PII) related to SPDC users, the CP and SMSS that will be contained within the SPDC. The SORN was submitted in March 2013, and the PIA was submitted in November 2013.
Computer Assisted Telephone Interviewing (CATI) Technology
The Training Utilization and Preservation Survey (TUP-S): State/Tribal and Campus Versions will be administered over the telephone using CATI technology. The evaluation division of ICF Macro, Public Health and Survey Research (PHaSR), operates fully integrated call centers in Burlington, Vermont, Seattle, Washington, and Martinsville, Virginia. These three centers currently offer a total of 340 CATI stations that conduct over 600,000 interviews annually. These centers are networked with each other. The facilities use the same CATI software, operate on the same platform, and are connected by a high-speed link that allows projects managed at one site to be conducted from the other site, or from both sites simultaneously. The CfMC questionnaire programming language provides call management and quota controls, inbound calling capabilities, multilingual interviewing capabilities, data back-up and monitoring, and incidence tracking. All of these CATI stations are equipped with predictive dialing capabilities. The use of PHaSR’s CATI system, predictive dialing system, and supervisory staff ensure that this data is methodologically consistent with other study efforts.
SMSS Technology
The Training Utilization and Preservation Survey (TUP-S): Adolescent Version and Short Message Service Survey (SMSS) will be administered via SMS (text messaging) technology. Text messaging is increasingly popular among younger Americans, as such, this medium is increasingly being adapted for research purposes, although the field is in its infancy. The medium has some limitations: questionnaires must be shorter (to limit break-offs or incomplete surveys), and questions must be shortened and reworded to fit within the SMS frame work (e.g., character limitations, brief instructions). ICF Macro has selected an external vendor to implement the survey and adapted the TUP-S Adolescent Version and the SSMS to fit the medium.
The cross-site evaluation team, in developing the data collection activities for the cross-site evaluation, conducted a literature review to avoid duplication in data collection activities and the use of similar information. Specifically, existing research studies and the efforts of other Federal initiatives designed to evaluate suicide or suicide prevention were reviewed.
Many in the field of suicide prevention agree that there is insufficient information on the causes of suicide and even less information on how to most effectively prevent suicide (SPAN USA, Inc., 2001; Institutes of Medicine, 2002; U.S. Public Health Service, 2001). The studies on suicide prevention activities have provided important information, but for the most part have been conducted with specific populations under certain circumstances and are not generalizable to other populations (Institutes of Medicine, 2002). Similarly, the lack of longitudinal and prospective studies has been a barrier to understanding and preventing suicide (Institutes of Medicine, 2002). Acknowledging the dearth of information on the effectiveness of suicide prevention programs, the Institutes of Medicine’s Report, “Reducing Suicide: A National Imperative” provides several recommendations for increasing research on suicide (2002). The report recommends that Federal funding be provided for the development, testing, and expansion of suicide prevention interventions, and for longitudinal studies that focus on the medium to long-term impacts of suicide prevention activities, such as the impact on risk and protective factors, treatment, and prevention. Specifically, the report recommends exploring the impact of suicide prevention programs through large nationally coordinated efforts.
Although there have been evaluations examining the effectiveness of specific suicide prevention activities, such as gatekeeper trainings, suicide screening programs, and skills trainings, these studies have focused on specific populations, mostly school-based, and have not assessed the impact of programs across multiple sites or across time (Eggert et al., 1997; King & Smith, 2000; Eggert, Nicholas & Owen, 1995). For example, an evaluation of the Lifelines School-Based Adolescent Suicide Prevention Program found increases in knowledge and help-seeking behaviors (Kalafat & Elias, 1994), but was specific to youth in schools. The cross-site evaluation will assess suicide prevention approaches across multiple sites targeting diverse youth groups to determine the impact of suicide prevention activities and the extent to which funded activities meet the goals and objectives of the GLSMA. Cross-site evaluation data will also be used to assess performance across time in these diverse settings in efforts to improve and enhance suicide prevention programs for funded and future funded grantees.
The existing knowledge base focuses on short-term impacts, and little is known about medium to long-term impacts of suicide prevention programs across broader and more diverse populations, as well as any direct impact on youth being referred for services. No evaluations have been conducted to examine the impact of suicide prevention programs across multiple sites with diverse populations, involving diverse child-serving agencies (i.e., mental health, juvenile justice, foster care), or to examine the impact on receipt of services. The cross-site evaluation of the GLS Suicide Prevention Program will present a unique opportunity to collect information from multiple sites implementing suicide prevention activities in efforts to assess the effectiveness of those activities and the impact on youth at risk for suicide. The information learned from previous research on suicide prevention activities was crucial in designing the cross-site evaluation but the cross-site evaluation does not include data collection activities that will collect similar information as previous studies.
The Centers for Disease Control and Prevention’s (CDC) National Center for Injury Prevention and Control provides funding and TA to states through its Core Violence and Injury Prevention Program (Core VIPP). The program supports 20 state health departments in strengthening their capacity to collect data and use data for a better understanding of local injury issues, including suicide. The focus of Core VIPP is on supporting funded state partners in their efforts to build a solid violence and injury prevention infrastructure, collect and analyze data, and implement and evaluate injury prevention programs. This CDC program may provide a broader understanding of suicide as a by-product of its efforts to gain a better understanding of local injury issues, however, the focus of the GLS cross-site evaluation is specifically to evaluate the effectiveness of suicide prevention programs.
SAMHSA is sponsoring an ongoing evaluation of the National Suicide Prevention Lifeline, the national crisis hotline. The purpose of the evaluation is to assess the impact of the national crisis hotline connecting callers to mental health professionals and to assess participation with the Lifelines networks. The specific focus of the ongoing evaluations changes as the components and dimensions of the National Suicide Prevention Lifeline expand. Although the data collection activities planned as part of this effort will provide valuable information on the effectiveness of this important service for at-risk youth, the scope of the evaluation focuses on all callers (adult and youth) to the national hotline and is specific to one intervention. The cross-site evaluation will add to the information collected as part of this effort to assess other suicide prevention strategies (i.e., gatekeeper training, suicide screening activities) and focuses on youth specifically.
Some of the data for this evaluation will be collected from individuals involved with public agencies, such as mental health, juvenile justice, education, and child welfare agencies and from colleges and university. While most data will be collected from public agencies or universities, it is possible that organizations involved in the referral networks would qualify as small entities. Also, respondents to the Training Exit Survey and the follow-up training qualitative interview, while most likely employed by public agencies, may also be employed by small businesses or other small entities. But these data collection activities will not have a significant impact on these agencies or organizations.
Cross-Site Evaluation
Product Stage. Grantees will be required to first complete the Baseline version of the Prevention Strategies Inventory (PSI) (see Attachments B.1 and B.2) in year 1 of the grant. Thereafter, they will be required to complete the Follow-up version of the Prevention Strategies Inventory on a quarterly basis over the duration of their 3-year grant period. Collecting this information quarterly is necessary to track progress toward meeting suicide prevention goals and to provide information on the development stage of products and services within State/Tribal and Campus programs. The consequences of collecting those data less frequently are the potential of losing information related to the process of developing and implementing products and services as well as losing the ability to track progress over time.
Process Stage. Both Campus and State/Tribal grantees are required to report aggregate training participant information for all trainings conducted as part of their suicide prevention programs in the format of the Training Activity Summary Page (TASP) (see Attachments H.3 and H.4). Since gatekeeper training is a widely implemented suicide prevention strategy among State/Tribal and Campus grantees, aggregate basic information about trainings is necessary to understand how grant funds are being utilized in support of training.
The Training Exit Survey (TES) Individual Form (see Attachments C.1–C.4) assesses participants’ training experiences immediately following the training and is collected one time at the conclusion of the training for Campus grantees. For a random sample of training participants from both State/Tribal and Campus grantees, the Training Utilization and Preservation Survey (TUP-S): State/Tribal, 6- Month Follow-Up, Campus and Adolescent versions (see Attachments D.1, D.4, D.7, and D.10) will be implemented 3 months following the training to collect information on the utilization of the knowledge, skills, and techniques learned through the training. The consequence of not collecting the training experience data at the conclusion of the training experience would be the absence of understanding and cross-site knowledge about the types of trainings being provided with grant funds, the quality of those trainings, and the individuals being trained. The consequences of not conducting the follow-up training utilization and preservation surveys and interviews would be a lack of important information concerning the impact and penetration of the suicide prevention training activities.
The Referral Network Survey (RNS) and the Coalition Survey (CS) (see Attachments E and G) will be administered to referral networks and coalitions identified by State/Tribal grantees once in the first year of the grant, and again during the third year of the grant. Two administrations of the Referral Network Survey and the Coalition Survey are important in learning whether the suicide prevention programs have an impact on building referral networks for youth identified as at risk for suicide. The consequences associated with less frequent data collection would be a lack of information that assesses the impact of time on the development of referral networks and coalitions. The Coalition Profile (CP) (see Attachment F) will be administered once during the grant.
For Campus grantees, the Short Message Service Survey (SMSS) (see Attachment K) will be administered once in the first year of the grant, and again in the third year of the grant. Data collected cross-sectionally at multiple points in time is necessary to assess any change in awareness and knowledge as a result of suicide prevention activities. If data were collected at only one time, there would be no means by which to assess change over time, an important element of the suicide prevention program.
The Student Awareness Intercept Survey (SAIS) (see Attachment J.1– J.2) will be implemented at up to four Campuses. The survey will be administered once at baseline, and then again 3 months after the baseline administration. It is important to collect this follow-up information in order to assess the reach and penetration of campus suicide prevention outreach and awareness initiatives, and the impact of these initiatives over time. In particular, the SAIS assesses participants’ knowledge of resources and referral/use of services, stigma related to mental health services, knowledge of suicide and its prevention, and help seeking behaviors.
For Campus grantees, the Life Skills Activities Follow-up Interview (LAFI) (see Attachment I.1) will be administered to respondents 3 months after their participation in life skills and wellness training activities targeted specifically for students on campus. Collecting the follow-up interview 3months after the training allows adequate time for students to have utilized the knowledge and skills learned through these life skills and wellness activities.
Impact Stage. To assess the impact of State/Tribal program activities, existing information on youth referred for services and service receipt as a result of early identification activities is analyzed. The Early Identification, Referral and Follow-up Analysis (EIRF) (see Attachment H.1) requires State/Tribal grantees to share existing data with the cross-site evaluation team on the youth identified as at risk as a result of early identification activities supported by their suicide prevention programs, their referral for services, and service receipt. State/Tribal grantees are also required to report aggregate screening information for all youth screened as part of their suicide prevention programs in the format of the Early Identification, Referral and Follow-up Screening Form (EIRF-S) (see Attachment H.2). To assess the impact of Campus program activities, the cross-site evaluation team will request campus sites to engage in a MIS Data Collection Activity (see Attachments H.5 and H.6) to submit existing administrative data related to the number of students who are at risk for suicide, the school retention rate, the number who seek services, and the type of services received, including emergency services. These data are collected by extracting existing data from several different sources, and are requested every quarter The consequences of not collecting this information will be lack of understanding of the impact of the suicide prevention program on the identification of youth at risk, their referral to services and their service receipt. Information tracked through these data collection activities is needed to report on proposed NOMs related to access to services and use of social supports as well as for GPRA reporting.
The data collection fully complies with the requirements of 5 CFR 1320.5(d) (2).
a. Federal Register Notice
SAMHSA published a notice in the Federal Register, volume 78, page 36205 on April 17, 2013, soliciting public comment on this study. SAMHSA received a public comment on August 13, 2013 regarding the planned data collection. The comment received proposed changes to questions about gender and sexual orientation. SAMHSA has reviewed the suggested changes and modified the instruments where the changes were appropriate.
b. Consultation Outside the Agency
Consultation on the design, instrumentation, and statistical aspects of the evaluation has occurred with individuals outside of SAMHSA. An evaluation steering committee was established in 2005 to provide input and guidance in designing and implementing the cross-site evaluation. Consultation with the evaluation steering committee began in 2005 and will continue as needed throughout the grant-funding period. Representatives on the steering committee include leaders in the field of suicide prevention and evaluation. In addition, representatives of the Suicide Prevention Resource Center (SPRC) were consulted with respect to the design of the cross-site evaluation in 2005. The SPRC provides TA to entities implementing suicide prevention programs. Input from representatives of the CDC was also solicited in 2005. The CDC has conducted research in the field of suicide prevention and was consulted to comment on the cross-site evaluation design, frequency of data collection activities, and instrumentation.
In addition, updates to the cross-site evaluation instruments were informed through direct consultation with current and former grantees, as well as representatives of the SPRC and CDC. These consultations had four purposes: (1) to ensure continued coordination of related activities, especially at the Federal level; (2) to ensure the rigor of the evaluation design, the proper implementation of the design, and the technical soundness of study results; (3) to verify the relevance and accessibility of the data to be collected; and (4) to minimize respondent burden.
Remuneration is a standard practice on university campuses, and has proven to increase response rates for college student surveys. In a study examining response rates in the National Survey of College Graduates, incentives provided to an experimental group resulted in an increase in response rates of nearly 11% versus no incentives (Dillman, 2000).
Remuneration will be used for the Short Message Service Survey (SMSS) (see Attachment K), the Student Awareness Intercept Survey (SAIS) (see Attachments J.1 and J.2), the Training Utilization and Preservation Survey (TUP-S): State/Tribal, Campus, and Adolescent Versions (See Attachments D.1, D.4, and D.7), and the Life Skills Activities Follow-up Interview (LAFI) (see Attachment I.1). Payment will not be provided to any other respondents as part of the cross-site evaluation. Respondents to other data collection activities are primarily staff of the suicide prevention programs or close affiliates. Therefore, no remuneration is planned for those activities.
Incentives have been historically utilized with 3 data collection activities of the cross-site evaluation, including the qualitative Training Utilization and Preservation Interviews (TUP-I) ($20 money order), Training Utilization and Preservation Surveys (TUP-S) ($10 money order or electronic Amazon gift code) and the SPEAKS ($5 incentive). We have proposed to continue providing a $10 incentive for participants in all TUP-S versions implemented as part of the evaluation. We have also proposed to provide a $20 incentive for Life Skills Activities Follow-up Interview (LAFI) participants; as this activity is a qualitative interview similar to the TUP-I. The incentive for the TUP-S and LAFI is introduced during the consent-to-contact process. The consent-to-contact process assists in ensuring representative members of trainings participate in the survey.
An incentive study for the SPEAKS campus evaluation activity was conducted in 2012 (Findings were presented to the American Evaluation Association Annual Conference in 20122 and are being prepared for manuscript submission. Study findings indicated that utilizing some form of incentive in Web-based surveys of college students yielded significantly higher response rates than not using any form of incentive. In addition, we found that across all campuses, using $5 incentives yielded significantly higher response rates than lottery incentives. The findings are consistent to general findings related to the use of some form of incentive versus no incentive. Given these results we are continuing to propose the use of incentives in the SMSS ($5 incentive) and SAIS ($10 incentive) data collection activities.
Short Message Service Survey (SMSS)
Based on our experience implementing similar surveys with Campus grantees, this survey will be implemented using short message survey (SMS) or text messaging technology in order to increase response rates. Respondents will receive an initial text message asking if they want to participate in a text survey about suicide prevention. Upon responding “yes,” participants will receive 2–3 messages containing consent language and information to contact the National Suicide Prevention Hotline if they need help. Upon consent to participate, respondents will receive a $5 Amazon gift code embedded in a text message. Once received, this code is good indefinitely.
Student Awareness Intercept Survey (SAIS)
Based on our experience implementing similar surveys with Campus grantees, including feedback from participants, a mixed mode approach, as well as an incentive, will be used to increase response rates. The baseline SAIS will be administered by ICF Macro’s staff onsite at up to four participating campuses. Respondents will receive a $5 gift card or cash incentive for their participation. After participation, respondents will be asked to provide their email address and phone number for participation in a 3-month, follow-up survey. Participants for the 3-month, follow-up survey will be contacted via email and telephone. Those who participate in the 3-month, follow-up interview will receive a $5 online gift code.
Training Utilization and Preservation Survey (TUP-S) and Life Skills Activities Follow-up Interview (LAFI)
Remuneration is a standard practice in longitudinal studies in efforts to maintain participation in the study. Recontacting survey respondents for follow-up interviews is difficult given the lapse in time between the original survey and the follow-up interview. Compounding the difficulty is respondents who are not directly affiliated with the programs being evaluated. Therefore, given the hard-to-reach nature of these populations, an incentive will be provided for two cross-site evaluation data collection activities that involve follow-up interviews. Participants in the TUP-S State/Tribal, 6-Month Follow-Up, Campus and Adolescent Versions will receive a $10 money order or electronic gift code incentive. Participants in the LAFI, will receive a $20 money order or electronic gift code incentive. An incentive for these respondents is particularly deemed appropriate because they are gatekeepers not directly affiliated with the suicide prevention program.
A Web-based data collection and management system was designed to facilitate data entry and management for the cross-site evaluation. Descriptive information will be collected from respondents to cross-site evaluation data collection activities, but no identifying information will be entered or stored into the Web-based data collection and management system. Identifying information will be requested in order to facilitate the Training Utilization and Preservation Survey (TUP-S): State/Tribal, 6-Month Follow-Up, Campus and Adolescent Versions, the Referral Network Survey (RNS), the Coalition Survey (CS), the Life Skills Activities Follow-up Interview (LAFI), the Student Awareness Intercept Survey (SAIS), and the Short Message Service Survey (SMSS). Identifying information will not be stored with survey responses and specific procedures to protect the privacy of respondents are described below for each data collection activity.
Prevention Strategies Inventories (PSI). Information to complete the inventories will be directly entered into the Web-based system. To access the system, respondents receive an individual username and password to protect their privacy and no identifying information is requested on the inventories.
Training Exit Survey (TES) Individual Form. Each respondent to the Training Exit Survey Individual Form will be provided a randomly generated training participant ID, but no identifying information will be requested on the survey. Responses to the survey will be entered into the Web-based system, but no identifying information will be entered.
Training Utilization and Preservation Survey (TUP-S): State/Tribal, 6-Month Follow-up, and Campus Versions. Contact information for the telephone-administered TUP-S will be collected through the consent-to-contact forms that will be distributed at training events. The consent-to-contact form will include a training participant ID (which contains no identifying information) and will ask participants to provide the identifying information (name, telephone number, and mailing address) necessary for contacting them for the TUP-S and for administering the incentive. The hard copy consent-to-contact forms will be stored in locked cabinets and the contact information will be entered into a password-protected database that can only be accessed by the limited number of individuals (selected ICF Macro staff such as telephone interviewers, data analysts and administrative staff for administering the incentives) who require access. These individuals have signed privacy, data access and use agreements. Datasets used by the data analysts will be stripped of any identifying information. Once the incentives are sent out, respondent contact information will be deleted from the database and the hard copy consent-to-contact forms will be destroyed. At the start of the telephone interview, verbal consent will be obtained from the respondents (see Attachments D.2, D.5, D.6, and D.8).
Training Utilization and Preservation Survey (TUP-S): Adolescent Version. Consent for an adolescent’s participation in a training activity will be collected from their caregiver/guardian and from the adolescent by the grantee program staff or training facilitator prior to the training. ICF Macro will work with grantees to collect parental and youth consent for the youth participation in the TUP-S. Additionally, contact information necessary for administering the TUP-S and the incentive will be collected via the consent-to-contact form (see Attachment D.8) that will also be distributed with the consent for participation in the training event by the grantee and training coordinator. The consent-to-contact form will include a training ID and ask participants to provide identifying information (name, cell phone number, and mailing address) necessary for contacting them for the TUP-S and for administering the incentive. As with the State/Tribal and Campus TUP-S, hard copy consent-to-contact forms and consent forms will be stored in locked cabinets and the contact information will be entered into a password-protected database which can only be accessed by the limited number of individuals (selected ICF Macro staff such as data analysts and administrative staff for administering the incentives) who require access. These individuals have signed privacy, data access, and use agreements. Datasets used by the data analysts will be stripped of any identifying information. Once the incentives are sent out, respondent contact information will be deleted from the database and the hard copy consent-to-contact forms will be destroyed. At the start of both the Web-based and text message versions of the survey, consent will be obtained from the respondents.
Referral Network Survey (RNS) and Coalition Survey (CS). Identifying information for respondents to the Referral Network Survey and the Coalition Survey will be necessary in order to administer the RNS and CS. Contact information will be limited to agency affiliations, names, email addresses and telephone numbers. Contact information will be entered into a password-protected database which can only be accessed by a limited number of individuals (selected ICF Macro staff such as telephone interviewers and cross-site team members) who require access. These individuals have signed privacy, data access, and use agreements. Datasets used by the data analysts will be stripped of identifying name and telephone number information. However, although the individual’s identifying name will not be used by any reports or datasets, the reports and datasets will contain the name of the agency/organization and the information provided about the agency or organization. Therefore, an individual may be identifiable when reporting results. Respondents are informed of possible identification in the consent language at the start of the Web-surveys for both the RNS and CS.
Short Message Service Survey (SMSS). Identifying information will be necessary to contact respondents via text message. Identifying information will be limited to a mobile phone number. Each Campus will provide a list of mobile phone numbers of all enrolled students. Each student whose number has been obtained will receive an initial text message asking if he or she would like to participate in a text survey about suicide prevention. Upon responding “yes,” the student will receive 2-3 messages containing consent language (see Attachment K), including background information about the study, risks and benefits, as well as information to contact the National Suicide Prevention Hotline if they require help. After the consent process, students will have the option to select “yes” to continue to the survey. Identifying information will not be stored or associated with survey responses, and will not be used for analysis or reporting efforts.
Student Awareness Intercept Survey (SAIS). The SAIS will be administered with up to four Campus grantees. Campuses implementing targeted suicide prevention campaigns will be identified and selected by reviewing grant applications and through technical assistance activities. Identifying information of students on the four participating campuses will not be necessary for recruitment of the baseline intercept survey. Contact information (e.g., name, address, email address) will be collected after completion of the baseline survey in order to conduct a 3-month, follow-up survey with baseline respondents. Three months after the baseline SAIS, respondents will be sent an email with a password and URL to access the follow-up survey. Identifying information will be limited to email addresses and phone numbers for recruitment purposes only and will not be stored or linked with survey responses. Contact information will be entered into a password-protected database which can only be accessed by the limited number of individuals (selected ICF Macro staff such as telephone interviewers, data analysts and administrative staff for administering the Web-based survey) who require access. These individuals have signed privacy, data access, and use agreements. Respondents will be assigned a unique password. To ensure privacy, no identifying information will be entered in the data collection and management system. Therefore, no identifying information will be associated with individual responses and no identifying information will be used for analysis or reporting efforts.
Life Skills Activities Follow-up Interview (LAFI). Campus program staff will distribute and collect consent-to-contact forms (See Attachment I.2) from trainees interested in being re-contacted for participation in the LAFI. Campus program staff will forward the completed consent-to-contact forms to the cross-site evaluation team. The consent-to-contact form will include identifying information necessary for contacting respondents; however, no identifying information will be entered into the Web-based data collection and management system and all consent-to-contact forms will be stored separately from LAFI responses in order to protect the privacy of respondents. For respondents that are not selected for participation, or who refuse to participate, their consent-to-contact forms will be destroyed upon completion of the study component. The hard copy consent-to-contact forms will be stored in locked cabinets and the contact information will be entered into a password-protected database which can only be accessed by a limited number of individuals (selected ICF staff such as telephone interviewers, data analysts, and administrative staff for administering the incentives) who require access. These individuals have signed privacy, data access, and data use agreements. Once the incentives are sent out, respondent contact information will be deleted from the database and the hard copy consent-to-contact forms will be destroyed. At the start of the telephone interview, verbal consent will be obtained from the respondents (see Attachment I.3).
Survey and interview instruments include questions that are potentially sensitive because this project concerns suicide prevention. These questions collect information about mental health, substance abuse, family circumstances, mental health services seeking, and suicide. These questions are central to the agency’s goal of learning about the protective factors and campus wellness context related to suicide prevention. Names and email addresses collected as part of the consent process will be kept separate from responses as stated above. All data will be managed and stored in the manner described above and therefore will be unavailable to anyone but authorized project staff. Active consent forms explicitly advise potential respondents and participants about the sensitive nature and content of the data collection protocol as well as the voluntary nature of all data collection activities. Unanticipated or negative consequences will be reported immediately to the campus and ICF Macro institutional review boards (IRB). The Principal Investigator and Project Director will also consult with appropriate clinical professionals and immediately determine if the participant presents a risk to themselves or others and make appropriate referrals. The proposed protocols were submitted to the ICF IRB for review and were approved on September 27, 2013. Updated protocols will be submitted to the IRB once OMB approval is received.
Data collection for the cross-site evaluation for the 61 State/Tribal grantees and the 60 Campus grantees will cover a 3-year project period. Data collection for the currently active grantees is operating under the previously approved OMB clearance, which expired in August 2013, and is currently extended through 2013 as the current package is under review.
Table 3 below shows the burden associated with cross-site evaluation data collection activities and the associated costs. The number of grantees for which burden is calculated is 121 (61 State/Tribal grantees and 60 Campus grantees), which represents the number of currently active grantees. It should be noted that this number is being used as an estimate of the number of grantees that are active per year. Fifty nine grantees (out of the 121 grantees) were funded in September 2011 and will reach the end of their grant period in September 2014. At that point, additional grantees may be funded. Therefore, it is estimated that, in a given year, there would be 121 active grantees. Table 4 shows an annualized summary of burden hours by respondent type.
The cost was calculated based on the hourly wage rates for appropriate wage rate categories using data collected as part of the National Compensation Survey (BLS, 2011) and from the US Department of Labor Federal Minimum Wage Standards.
Table 3. Estimated Annual Burden Hours and Costs
Note: Total burden is annualized over the 3-year clearance period.
State/Tribal Cross-Site Evaluation Instruments |
||||||||
Type of Respondent |
Instrument |
Number of Respondents |
Responses per Respondent |
Total Number of Responses |
Burden per Response (hours) |
Annual Burden (hours) |
Hourly Wage Rate ($) |
Total Cost ($) |
Project Evaluator |
Prevention Strategies Inventory - State Tribal (PSI-ST) |
61 |
4 |
244 |
0.75 |
183 |
37.82 |
6,922 |
Provider (Trainees) |
Training Utilization and Preservation Survey (TUP-S) |
2,000 |
1 |
2,000 |
0.16 |
320 |
21.35 |
6,832 |
Adolescents (Trainees) |
Training Utilization and Preservation Survey (TUP-S) |
300 |
1 |
300 |
0.16 |
48 |
7.25 |
348 |
Provider (Trainees) |
Training Utilization and Preservation Survey (TUP-S): 6-Month Follow-up |
467 |
1 |
1,467 |
0.16 |
75 |
21.35 |
1,602 |
Provider (Stakeholder) |
Referral Network Survey (RNS) |
1,426 |
1 |
1,426 |
0.67 |
956 |
21.35 |
20,411 |
Project Evaluator |
Coalition Profile (CP) |
33 |
1 |
33 |
0.33 |
11 |
37.82 |
417 |
Provider (Stakeholder) |
Coalition Survey (CS) |
426 |
1 |
426 |
0.67 |
286 |
21.35 |
6,107 |
Project Evaluator |
Early Identification, Referral and Follow-up Analysis (EIRF) |
61 |
4 |
244 |
5 |
1,220 |
37.82 |
46,141 |
Project Evaluator |
Early Identification, Referral and Follow-up Screening Form (EIRF-S) |
27 |
4 |
108 |
1 |
108 |
37.82 |
4,085 |
Project Evaluator |
Training Activity Summary Page (TASP-ST) |
61 |
4 |
244 |
.33 |
81 |
37.82 |
3,064 |
Table 3. Estimated Annual Burden Hours and Costs (continued)
Note: Total burden is annualized over the 3-year clearance period.
CAMPUS CROSS-SITE EVALUATION INSTRUMENTS |
|||||||||
Type of Respondent |
Instrument |
Number of Respondents |
Responses per Respondent |
Total Number of Responses |
Burden per Response (hours) |
Annual Burden (hours) |
Hourly Wage Rate ($) |
Total Cost ($) |
|
Project Evaluator |
Prevention Strategies Inventory-Campus (PSI-C) |
60 |
4 |
240 |
0.75 |
180 |
37.82 |
6,808 |
|
Provider (Trainees) |
Training Exit Survey Campus (TES-C) |
4,333 |
1 |
4,333 |
0.17 |
737 |
21.35 |
16,375.45 |
|
Student |
Student Awareness Intercept Survey (SAIS) |
1,600 |
2 |
3,200 |
1 |
3,200 |
7.25 |
23,200 |
|
Student |
Short Message Service Survey (SMSS) |
5,200 |
1 |
5,200 |
0.083 |
432 |
7.25 |
3,132 |
|
Student |
Life Skills Activities Follow- up Interview (LAFI) |
35 |
1 |
35 |
0.50 |
18 |
7.25 |
131 |
|
Student |
Training Utilization and Preservation Survey (TUP-S) |
367 |
1 |
367 |
0.16 |
59 |
7.25 |
428 |
|
Project Evaluator |
Training Activity Summary Page Campus (TASP-C) |
60 |
4 |
240 |
0.33 |
80 |
37.82 |
3,026 |
|
Project Evaluator |
MIS Data Collection Activity |
60 |
1 |
240 |
0.33 |
20 |
37.82 |
757 |
|
TOTAL |
|||||||||
Total |
11715
|
-- |
13855
|
-- |
4726
|
-- |
$53,857 |
||
National Compensation Survey: Occupational Wages in the United States (2011, May). US Bureau of Labor Statistics (BLS) US Dept. of Labor. The category Social Scientists and Related Workers under Life, Physical and Social Science Occupations was used as an approximation for Project Evaluators.
Link: http://www.bls.gov/oes/current/oes_nat.htm#19-0000
National Compensation Survey: Occupational Wages in the United States (2011, May). US Bureau of Labor Statistics (BLS) US Dept. of Labor. The category Child, Family and School Social Workers under Life, Physical and Social Science Occupations was used as an approximation.
Link: http://www.bls.gov/oes/current/oes_nat.htm#19-0000
Federal Minimum Wage
Link: http://www.dol.gov/elaws/faq/esa/flsa/001.htm
Table 4. Annualized Summary Table
STATE/TRIBAL CROSS-SITE EVALUATION INSTRUMENTS |
||||
Respondents |
Number of Respondents |
Responses/ Respondent |
Total Responses |
Total Annualized Hour Burden |
Project Evaluators |
243 |
17 |
873 |
1,603 |
Adolescents (Trainees) |
300 |
1 |
300 |
48 |
Provider (Trainees) |
2,467 |
2 |
3,467 |
395 |
Provider (Stakeholder) |
1,852 |
2 |
1,852 |
1,242 |
CAMPUS CROSS-SITE EVALUATION INSTRUMENTS |
||||
Respondents |
Number of Respondents |
Responses/ Respondent |
Total Responses |
Total Annualized Hour Burden |
Project Evaluators |
180 |
9 |
720 |
280 |
Students |
7,202 |
5 |
8,802 |
3,709 |
Provider Trainees |
4,333 |
1 |
4,333 |
737 |
|
|
TOTAL |
|
|
Total |
16577 |
37 |
20347 |
8,014 |
Grantees are collecting the majority of the required data elements as part of their normal suicide prevention program operations. Grantees will maintain this information for their own program planning, quality improvement, and reporting purposes. Therefore, there are no additional capital or start-up costs associated with the cross-site evaluation. There will be some additional burden on record keepers to provide potential respondent lists for data collection activities. However, these operation costs will be minimal.
Other costs related to this effort, such as the cost of shipping completed questionnaires (i.e., training exit survey) and consent-to-contact forms is cost to the Federal Government as part of the funding received for participation in the cross-site evaluation. Each grantee has been funded, as part of the overall cooperative agreement award, to fund an evaluator and related costs to carry out the requirements of the cross-site evaluation. Therefore, no cost burden is imposed on the grantee by this additional effort.
CMHS has planned and allocated resources for the management, processing and use of the collected information in a manner that shall enhance its utility to agencies and the public. Including the Federal contribution to local grantee evaluation efforts, the contract with the National Evaluator, and Government staff to oversee the evaluation, the annualized cost to the Government is estimated at $4,092,003. These costs are described below.
Each grantee is expected to fund an evaluator to conduct the self-evaluation and to satisfy the requirements of the cross-site evaluation. It is estimated that participating in the cross-site evaluation will require 0.20 full-time equivalent (FTE) to collect information, enter information into the Web-based data collection and management system, and to conduct analyses at the local level. Assuming: 1) an average annual salary of $78,670 (BLS, 2011) for a 0.20 FTE evaluator; 2) 61 State/Tribal and 60 Campus grantees; and 3) that Campus grantees had to cost share on a 1:1 basis, the annual cost for the cross-site evaluation at the grantee level is estimated at $1,431,794. These monies are included in the cooperative agreement awards.
The cross-site evaluation contract has been awarded to ICF Macro for evaluation of the 121 suicide prevention programs. The current cross-site evaluation contract with SAMHSA provides $11,825,774 for a 5-year period. The estimated average annual cost of the contract will be $2,365,154. This covers expenses related to developing and monitoring the cross-site evaluation including. They include but are not limited to: developing the evaluation design; developing the cross-site evaluation instrumentation; developing training and TA resources (i.e., manuals, training materials, etc.); conducting in-person or telephone training and TA; monitoring of grantees; traveling to grantee sites and relevant meetings; and analyzing and disseminating data activities. In addition, these funds will support the development of the Web-based data collection and management system and fund staff support for data collection. It is estimated that CMHS will allocate 0.30 of a full-time equivalent each year for Government oversight of the evaluation. Assuming an annual salary of $80,000, these Government costs will be $24,000 per year.
Currently there are 26,444 burden hours in the OMB inventory. SAMHSA is requesting 8,014 hours for this submission. This represents a decrease in burden of 18,430 hours. The number of grantees for which burden is calculated is 121 (61 State/Tribal grantees and 60 Campus grantees), which represents the number of currently active grantees.
Major program changes that account for the change in burden are described below:
The previously approved Training Exit Survey State/Tribal (TES-ST) will no longer be administered to State/Tribal grantees. The estimated burden for this effort was 16,125 hours. The estimated burden for the campus grantees to administer the TES to their trainees is 737 hours per year.
The previously approved Campus Infrastructure Interviews (CIFI) will no longer be administered to Campus grantees. The estimated burden for this effort was 144 hours.
The 3 previously approved instruments collected by a subset of Campus grantees have been removed. The estimated burden for this effort was 542 hours.
The previously approved Training Utilization and Preservation Interview (TUP-I) will no longer be administered to State/Tribal grantees. The estimated burden for this effort was 67 hours.
It is proposed to implement the Training Utilization and Preservation Survey (TUP-S): Campus Version with Campus grantees in order to significantly increase our understanding of how training information is utilized. Campus sites were not previously required to do this survey. The estimated burden for this effort is 59 hours.
It is proposed to implement the Training Utilization and Preservation Survey (TUP-S): Adolescent Version to youth under the age of 18 who participate in trainings sponsored by State/Tribal grantees. This will significantly increase our understanding of how youth retain and implement information used from prevention trainings. Youth training participants were not previously required to do this survey. The estimated burden for this effort is 48 hours.
It is proposed to implement a Training Utilization and Preservation Survey (TUP-S): State/Tribal Version 6-Month Follow-up in order to enhance our understanding of the knowledge retained from suicide prevention trainings. The estimated burden for this effort is 75 hours.
It is proposed to implement a new data collection instrument: the Life Skills Activities Follow-up Interview (LAFI) with students who participate in life skills and wellness activities sponsored by Campus grantees. This survey will expand our knowledge of how students utilized skills and knowledge learned from trainings aimed at enhancing protective factors. The estimated burden for this effort is 18 hours.
It is proposed to implement two new data collection activities that will support the previously approved Referral Network Survey (RNS): the Coalition Profile (CP) and the (Coalition Survey). The Coalition Profile is a brief survey administered to State/Tribal grantees who report engaging in coalition-building activities. It will provide a profile of the coalition’s mission and structure. The Coalition Survey complements the RNS in that it collects information directly from coalition members about collaboration efforts. Coalitions often form a central part of a grantee’s prevention program, and this information has not previously been collected. The estimated burden for the Coalition Profile is 11 hours. The estimated burden for the Coalition Survey is 286 hours.
The previously approved Suicide Prevention, Exposure and Awareness Knowledge Survey for Students and Faculty (SPEAKS-S and SPEAKS-F) will be replaced by the Short Message Service Survey (SMSS) in all campuses and complemented by the Student Awareness Intercept Survey (SAIS) in up to four selected campuses. These instruments collect information similar to the SPEAKS, but in a more user-friendly and less burdensome manner. The estimated burden for the SMSS is 432 hours. The estimated burden for the SAIS is 3,200 hours. The estimated burden for the SPEAKS Student and Faculty Versions was 3,667 hours.
The time schedule for implementing the cross-site evaluation is summarized in Table 5. A 3-year clearance is requested for this project.
Table 5. Time Schedule
Begin data collection for 121 grantees (61 State/Tribal Grantees & 60 Campus Grantees) |
September 2013 (1 month after OMB approval estimated to occur in August 2013) |
Data collection completed for the grantees funded in FY2011 (A new cohort of grantees may replace this cohort of grantees) |
September 2014 |
Final GLS Campus and State/Tribal Programs Evaluation Report |
October 2014 |
Data collection completed for grantees funded in FY2012 (A new cohort of grantees may replace this cohort of grantees) |
September 2015 |
Final GLS Campus and State/Tribal Programs Evaluation Report |
October 2015 |
Data collection continues until expiry of OMB approval |
August 2016 |
The GLSMA requires annual reports summarizing the results of the cross-site evaluation. The cross-site evaluation team will analyze data collected and prepare interim annual reports to summarize key findings. A final report on the results of the cross-site evaluation is also required by the GLSMA, and will be produced by the cross-site evaluation team no later than 3 years after the grants were received.
Because of the importance of the cross-site evaluation to the field of suicide prevention, in collaboration with SAMHSA and the Government project officer, the results of the cross-site evaluation will be published in relevant professional journals to inform the research community as well as the decision making of policymakers and program administrators. Up to five publications are being considered, and will most likely be submitted in the final year of the cross-site evaluation. Program and the key findings, as well as manuscripts reporting results from the following data collection activities: Training Exit Survey, Training Utilization and Preservation Survey, Referral Network Survey, and the Early Identification, Referral and Follow-up form. All publications will be submitted to the Contracting Office Representative (COR) in draft form for review and approval prior to submission to the selected journal.
Examples of journals that will be considered as vehicles for publication include the following:
American Journal of Public Health
American Psychologist
American Journal of Diseases of Children
Child Development
Crisis
Evaluation Review
Evaluation Quarterly
Journal of the American Academy of Child and Adolescent Psychology
Journal of Applied Development Psychology
Journal of Child and Family Studies
Journal of Clinical Child and Adolescent Psychology
Journal of Consulting and Clinical Psychology
Journal of Health and Social Behavior
Journal of Mental Health Administration
Psychological Reports
Social Services Review
Suicide and Life Threatening Behavior
Cross-site evaluation data collected through the different stages of the evaluation will be analyzed to address key evaluation questions. Table 6 displays the core process and outcome evaluation question, as well as key indicators and their respective source/data collection instrument. While the evaluations questions cut across different types of prevention activities, the indicators and instruments are generally specific to each type. The emphasis in this table is the connection of questions, indicators and instruments with the overall program logic model. Table 7 summarized the analytical approach for each evaluation question, providing examples of indicators and their respective source. The analytical approach for each question is described in more detail in the following subsection.
Cross-Site Evaluation
What types of prevention/intervention programs were used?
The Prevention Strategies Inventory (PSI) is the main source of information regarding the particular set of prevention strategies that each grantee implemented as well as the level of support in terms of GLS funds apportioned to each of them.
The cross-site evaluation team will compute summary descriptive statistics on the number of sites adopting each specific intervention and use robust statistics of central tendency and dispersion to summarize how sites apportion their budget to each type of intervention. Cross-tabulation by type of grantee (State, Tribe or Campus), funding cohort, and additional grantee characteristics (e.g., the size of the Campus) will be performed. Clustering techniques (such as k-means) are particularly useful to analyze patterns of budget allocation and will permit identification of groups of sites with similar focus. An alternative strategy would be to identify a discrete number of patterns or classes in the evolution of an outcome of interest, and then explore characteristics of the program that may predict class membership. This strategy is based on statistical models known as pattern-mixture models.
What were the process measures? What is the reach of the early intervention and prevention strategies?
The reach of the prevention strategies implemented by GLS grantees is assessed through different instruments specifically designed for each type of strategy. The reach of training strategies—encompassing both gatekeeper and assessment and referral trainings—is assessed through two instruments: the Training Activity Summary Page (TASP) and the Training Utilization and Preservation Survey (TUP-S). Life Skills development activities will be captured through the newly developed Adolescent TUP-S in the cases of State and Tribal grantees, while they will be explored through the Life Skills Activities Follow-up interview (LAFI) for Campuses. The Early Identification Referral and Follow-up Screening Form (EIRF-S) collects information on the reach of screening strategies. Exposure to suicide prevention efforts is broadly measured on Campuses through the Short Message Service Survey (SMSS), while exposure to specific social marketing campaigns is assessed with the Student Awareness Intercept Survey (SAIS).
The cross-site evaluation team will rely on descriptive statistics to determine the reach of these early intervention and prevention strategies. Descriptive summaries concerning the reach of the program such as the number of participants in gatekeeper trainings; professionals trained through assessment and referral trainings; youth receiving in-school or community early identification programming; youth participating in life skills development activities; and campus students exposed to awareness campaigns will be provided. Demographic characterization of these populations will also be performed, as well as cross-tabulation by other relevant characteristics, such as the setting of these prevention activities.
Training and early identification instruments (TASP, TUP-S, Adolescent TUP-S, EIRF-S) are administered on an ongoing basis, which allows the analysis to focus not only on the cross-sectional status but also to describe how the respective outputs change over time. Although SMSS and SAIS have discrete administration periods, both are administered twice during the life of the grant (first and third year of the grant, and before and after the social marketing campaign is implemented, respectively), which also supports comparable analyses.
Some of these instruments (namely TUP-S, SMSS and SAIS) are based on probabilistic samples of their respective target population. In such cases, appropriate measures of uncertainty (i.e., standard error and confidence intervals) will be computed and reported with their respective summary statistics. The appropriate measure of uncertainty depends both on the sample design and the target of inference. For SMSS in particular, inference targeting a particular cohort of Campus grantees should consider their difference in size and the different probability of selection resulting from the same size sample per campus design. On the other hand, model-based inference may ignore sample weights, but should take into account the correlation of observations within the same campus.
In the case of State/Tribal grantees, measuring the reach of the prevention activities typically faces the challenge of determining the relevant denominators. Indeed, catchment areas vary in size and rarely encompass the entire state. To address this issue, information on the location of the different prevention efforts (such as trainings, screenings, and identifications) are now systematically collected (at the level of the zip code) through several of the instruments mentioned above. This addition opens up the possibility of exploiting extant data, such as census information, to determine relevant denominators, such as the numbers of youth residing in the area where the prevention efforts take place.
What kinds of services were recommended to youth who were determined to be at risk for suicidal behavior?
The type of service recommended to youth determined to be at risk for suicidal behavior, together with additional follow-up information—such as whether the service was actually received within 3 months—is collected through two instruments: the EIRF form for State/Tribal grantees, and the TUP-S Campus Version. In the first case, the information refers to identifications resulting from formal screenings or those performed by gatekeepers acting on “closed,”’ well defined settings, with relatively well- developed information systems. The TUP-S captures follow-up information from gatekeepers that were trained by GLS grantees more generally, regardless of the setting of the identification or the level of development of the information system in place.
Descriptive statistics will be used to determine the types of referrals youth received, such as in-school counseling, community mental health, or emergency services. Chi-square tests and related analysis techniques will be used as appropriate to compare referral patterns across identification settings and source of referral. The likelihood of service receipt at follow-up as a function of site- and individual-level characteristics will also be examined within a mixed-effect regression framework. Both instruments are administered on an ongoing basis, which allows the analysis to incorporate changes over time in referral patterns.
What sorts of linkages were made as a result of the referral mechanisms used?
To understand the influence of referral mechanisms on subsequent linkages, it is proposed to use social network analysis on the information collected through the Referral Network Survey (RNS) and the Coalition Survey (CS). The two components of the redesigned survey focus on different types of linkages that are relevant to suicide prevention. The CS focuses on the linkages at a macro level, among organizations and agents that can support suicide prevention goals broadly across the grantee catchment area. The RNS component focuses on linkages at a micro level that are immediately relevant for the gatekeeper identifying a youth at risk to be able to connect the youth to the appropriate resource. The two levels also correspond to different types of interventions that grantees implement, such as coalition-building on the one hand, and policy and protocol development on the other.
Both surveys are administered twice during the cycle of the grant: early in the first and second years and late in the grant third year. This allows for the analysis to focus not only on the cross-section description of the status of the collaboration but also on the observed changes that might result from the grantee’s efforts. Social network analysis will examine the collaborations occurring between organizations within a potentially complex web of referral sources. Examining basic characteristics such as quality and symmetry of relationships, centrality, and density, will provide information on the extent to which the major agencies of youth-serving systems are working together to ensure that at-risk youth receive services.
What are the mediators associated with changes in outcomes of these programs and services?
Short and medium term outcomes of training and screening interventions (such as the number of youth identified at risk that were connected to the required service) are regularly assessed through the TUP-S and the EIRF. In addition, the SMSS and the MIS now collect suicidal behavior information (such as ideation and attempts) from Campuses, which constitute the final outcomes of the prevention program. Though suicidal behavior is not generally directly captured by State/Tribal instruments, measures derived from extant datasets (such as CDC’s National Vital Statistics System) can now be connected to suicide prevention efforts at a much more detailed geographical level, due to the inclusion of geographical location information in both the TASP and the EIRF.
The association of outcome indicators with both individual-level and site/program-level mediators will be examined using multivariate analysis techniques. In particular, it is proposed to use parametric modeling for outcomes of interest within a mixed-effect regression framework. As with other regression techniques, coefficients representing the differential importance of each mediator in predicting the outcome will be estimated. Unlike simple regressions, however, mixed-effect or multilevel models allow for correct statistical inference by accounting for the clustering of observations within the site. Furthermore, the models provide estimates of the relative importance of the source of variation. Finally, by borrowing strength from sites with a greater number of observations, Bayesian estimation of site random effects can be used to identify over- and underachieving sites with added precision, which in turn might suggest additional hypotheses regarding mediators.
The specific type of regression to be implemented will vary depending on the outcome under analysis. For instance, counts such as the number of children identified at risk, number of children referred to services, or the number of suicide attempts, are more naturally modeled using Poisson distribution, while the likelihood of receiving service at follow-up is more adequately modeled using logistic distribution.
Table 6: Process and Outcome Evaluation Questions, Key Indicators and Source |
||||
Process |
Outcomes |
|||
Evaluation questions |
||||
- What types of prevention/intervention programs were used? - What is the reach of the early intervention and prevention strategies? |
- What kinds of services were recommended to youth who were determined to be at risk for suicidal behavior? - What sorts of linkages were made as a result of the referral mechanisms used? - What are the mediators associated with changes in outcomes of these programs and services? |
|||
Key indicators [Source/Instrument] |
||||
Input |
Output |
Short term outcome |
Medium term outcome |
Long term outcome |
Awareness activities [PSI] |
# exposed [SMSS] |
Youth (at risk) /families are aware of available recourses and willing to use them (seek help) when needed [SAIS] |
Youth at risk are identified early and connected to appropriate service [EIRF] |
Reduce attempt, self-injury, suicide [SMSS; MIS; extant data (e.g. NVSS)] |
Gatekeeper Training [PSI] |
# GK trained [TASP] |
“Natural” gatekeepers are more capable of identification, referral and follow-up [TUP-S] |
Youth at risk are identified early and connected to appropriate service. [TUP-S; EIRF] |
|
Assessment & Referral Trainings [PSI] |
# professionals trained [TASP] |
Health professionals are more capable of identification, referral and follow-up [TUP-S] |
||
Screening Programs [PSI] |
# youth screened [EIRF-S] |
Youth at risk are identified early and connected to appropriate service. [EIRF-S] |
||
Life skills Development /Wellness activities [PSI] |
# youth participants [ TASP] |
Youth at risk are able to cope with stressors “constructively”; increased self-capacity to produce health [ Adolescent TUP-S; LAIF] |
||
Policies & Protocols for Intervention & Postvention [PSI] |
Protocol in place [CP-CS-RNS] |
Agencies involved in EIRF collaborate more effectively [CP-CS-RNS] |
Youth at risk are identified early and connected to appropriate service. [RNS; EIRF] |
|
Coalitions & Partnerships [PSI] |
# links created [CP-CS-RNS] |
|||
Table 7: Evaluation Questions, Indicators, Sources and Analytical approach |
||
Evaluation question |
Examples of Indicators [Main Source] |
Analytic approach |
What types of prevention/intervention programs were used? |
|
|
What is the reach of the early intervention and prevention strategies? |
|
|
What kinds of services were recommended to youth who were determined to be at risk for suicidal behavior? |
|
|
What sorts of linkages were made as a result of the referral mechanisms used? |
|
|
What are the mediators associated with changes in outcomes of these programs and services? |
|
|
All data collection instruments will display the expiration date of OMB approval.
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.
1 The evaluation as designed includes four stages (context, content, process, and impact) each of which is hinged to the fundable activities of the grantees, the research questions outlined in the evaluation statement of work, and the state of the knowledge base in the field of suicide prevention. As such, while the evaluation design does not currently include rigorous impact assessment, it does include the comparative assessment of proximal outcomes as a part of the impact stage. Hereafter, the impact stage is used as an umbrella term to cover evaluation protocols designed and implemented to understand the outcomes of the program.
2 Sgro, G., Xu, Y., Brewer, B., Rodi, M.S., Walrath, C.(2012, October). An Exploration of Strategies to Improve Web-based Survey Response Rates on College Campuses. Paper presented at the 26th Annual Conference of the American Evaluation Association.
Suicide
Cross-Site Evaluation Page
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-28 |