The respondent universe and sampling methods are described for each of the data collection instruments below.
Prevention Strategies Inventory (PSI). Respondents for the Prevention Strategies Inventory will be project evaluators and/or program staff. Each of the 61 State/Tribal grantees and 6 Campus grantees will be required to complete the inventory.
Training Exit Survey (TES).
The target populations of the TES is adult (i.e. 18 and older) participants in GLS Campus sponsored trainings in one of the 60 grantees from cohort 5 and 6 during FY 2014 to FY2016. It is estimated that these grantees will train as much as 45,000 adults during that period.
We propose a single-stage cluster sampling. Within each stratum, composed by a cohort of campus grantees, a simple random sample of grantees is selected. The TES is administered to all the adult trainees participating in training events in the selected campuses. When compared with other alternatives (such as simple random sampling of trainees or cluster sample of trainings), cluster sampling of campuses results in a reduction in the amount of information per sampled unit. However, the proposed alternative has several advantages, in particular, it minimizes the administrative costs and logistic requirements imposed to the grantees implementing the trainings, some of which may not have the capacity to carry out the procedures that alternative sampling strategies would require. For the same reason, the proposed alternative maximizes the likelihood that data collection is performed according to protocol.
A total of 30 grantees will be randomly selected, 10 from cohort 5 and 20 from cohort 6 stratum. We anticipate that, at a minimum, the proposed sampling plan will result in a total of 13,000 respondents (6, 5 and 2 thousand in fiscal years 2014, 2015 and 2016, respectively) based on a conservatively estimated response rate of 60%.
For the power analysis, we focused on a set of selected indicators, and compute the precision associated with their estimation (see Table 8). Along with the results for the overall sample, results are shown for domains representing 50%, 25% and 10% of the sample. In the context of TES, a typical domain of interest is constituted by the subtype of training or the main role of the participants. The main inputs for the analysis (i.e. mean, standard deviation and intra-class correlation for each indicator) are estimated based on more than 16,000 TES interviews from trainings implemented by cohort 4 and 6 sites between FY2012 and 2013. In sum, the overall sample affords quite precise estimations with standard errors no greater than 2.5%. For domains of moderate size (representing between 25% and 50% of the sample) the precision is still considerable with standard errors no greater than 5% unless the intra-class correlation (ICC) is unusually large. Only estimation in small domains (representing 10% of the sample) becomes relatively imprecise, particular when in presence of a large ICC.
Table 8 TES: Standard Error for selected indicators
Estimated Values |
Overall |
Domain |
||
50% of the sample |
25% of the sample |
10% of the sample |
||
Intent to use what was learned to identify students who might be at risk for suicide (Mean=.73; SD=.45; ICC=.06) |
1.4% |
2.9% |
4.1% |
6.5% |
Strongly Agree that the training increased their knowledge about suicide prevention (Mean=.60; SD=.49; ICC=.04) |
1.3% |
2.6% |
3.7% |
5.8% |
Trainee
involve in crisis response, primary or mental health services,
residential life services or teaching |
2.5% |
5.0% |
7.1% |
11.3% |
Training Utilization and Preservation Survey (TUP-S): State/Tribal, 6-Month Follow-up, Campus, and Adolescent Versions. The target population of TUP-S instruments is participants in GLS sponsored trainings. The different versions of the instrument target three distinct strata within that population. The State/Tribal TUP-S and TUP-S 6-month follow-up, will target adults (18 and older) who participated in State/ Tribal sponsored trainings (about 900 per grantee in FY 2012). The Adolescent TUP-S will target adolescents (12 to 17) who participated in State and Tribal sponsored trainings (approximately 170 per grantee in FY 2012). The campus TUP-S will target adults (18 and older) who participated in Campus sponsored trainings (about 450 per grantee in FY 2012).
All adult participants of GLS sponsored trainings will be administered a consent-to-contact form by the training facilitator or grantee staff during a training event. Respondents to the State/Tribal TUP-S will be asked to consent to be contacted for a second time (in 3 months). Consent to contact for the Adolescent TUP-S will be obtained from parent/guardians by training facilitators and/or grantee staff in conjunction with the consent to participate in the training itself.
The cross-site evaluation team will select a probabilistic sample of participants who consent to be contacted on an ongoing basis, as trainings are implemented and consents received, using systematic sampling. The sample fraction will be determined and updated yearly based on the projected number of consents so as to ensure the target sample sizes per year. Changes in the sample fraction will alter inclusion probabilities and must be taken into account in the analysis across years through the use of sampling weights.
Target sample sizes were determined so as to afford small standard errors for the estimates of the quantities of interest in a given year considering available resources. In addition, the sample size for each version is roughly proportional to the size of the stratum they represent in FY 2012. Key survey estimates will take the form of the percentage or proportions, such as the proportion of trainees who identified a youth at risk for suicide during the 3 months after the training. In the case of the TUP-S 6-month follow-up, the main interest is the change between administrations in these proportions of interest. Results are presented for the maximum standard errors, i.e., for a proportion close to 50%—in which the variance is the largest—and for no correlation over time in the case of the TUP-S 6-month follow-up. Along the result for the overall sample, results are shown for domains representing 50%, 25% and 10% of the sample. In sum the State/Tribal TUP-S has considerable precision even for estimation within small domains (e.g. maximum standard errors no greater than 3.5% for a domain comprising 10% of the sample). The new TUP-S instruments (i.e. State/Tribal TUP-S 6 months follow up; Campus TUP-S and Adolescent TUP-S), however, afford reasonable precision only for relatively large domains (comprising at least 50% of the sample). Nevertheless, increased precision for small domains can be obtained by combining two or three years of data collection.
Table 9. TUP-S instruments: Maximum Standard Error for a proportion
Instrument Version |
Overall |
Domain |
||
50% of the sample |
25% of the sample |
10% of the sample |
||
State/Tribal TUP-S |
1.1% |
1.6% |
2.2% |
3.5% |
State/Tribal TUP-S 6-Month Follow-up* |
2.9% (5.0% pilot) |
4.1% |
5.8% |
9.1% |
Campus TUP-S |
2.2% (5.0% pilot) |
3.2% |
4.5% |
7.1% |
Adolescent TUP-S |
2.5% (5.0% pilot) |
3.5% |
5.0% |
7.9% |
*Note the precision here is for a difference in proportions, instead of a single proportion, assuming no correlation over time. |
||||
Referral Network Survey (RNS). Using zip code data submitted by grantees on the Training Activity Summary Page forms, cross-site evaluation staff will determine the county or region where the grantee has the greatest impact. The grantee will then be asked to provide contact information for at least one and up to three organizations in this county or region. Cross-site evaluation staff will make a preliminary phone call to ask these primary organizations for their referral network. Using snowball sampling to determine the entire referral network for the county or region, cross-site evaluation staff will contact all organizations within the referral network to conduct the Referral Network Survey. Snowball sampling will be repeated until saturation is reached. However, in large networks, four waves with an average of three referrals per wave will be conducted, for a total of 27 respondents. For these large networks, protocol will be followed:
Wave 1—grantee identifies one respondent
Wave 2—1 agency provides 3 respondents
Wave 3—3 agencies each can provide 3 more respondents
Wave 4—9 agencies can each provide 3 respondents
If the participant agrees to participate in the survey during the initial phone call, respondents will be asked to provide a current email address. Once the referral network has been established, respondents will be sent an online survey. This online survey will be prefilled with the entire list of the network so respondents may select which organizations are in their direct referral network.
Coalition Survey. All grantees that indicate participation in coalition-building activities in the Prevention Strategies Inventory (PSI) will be asked to identify up to 10 members of their coalition. Respondents will be contacted initially by phone by the cross-site evaluation staff, and will be asked to provide an email address. Respondents will then be sent a link to the online survey.
Coalition Profile. All grantees who indicate their involvement with coalition-building efforts in the Prevention Strategies Inventory (PSI) will be contacted to complete a short survey about their primary coalition. If the grantee is involved with multiple coalitions, the grantee may select only one. Responses to the survey will be used to determine coalition members and basic information about the coalition structure.
Life Skills Activities Follow-up Interview (LAFI). Many of the Campus programs are planning multiple training activities; therefore, in attempts to obtain information from key informants who experienced the same training activity, the cross-site evaluation team, in consultation with local program staff, will select five particular training activities per year in which to administer the LAFI. Trainees will be asked to complete consent-to-contact form indicating their willingness to be contacted to participate in the LAFI and return the form to local program staff. Key informants for the LAFI will be randomly selected from those individuals who consent to be contacted by the cross-site evaluation team. Local program staff will forward the consent-to-contact forms to the cross-site evaluation team. Up to seven respondents from each of the five selected trainings will be randomly selected from among the potential respondents based on consent-to-contact information, for a total of up to 35 respondents per year. Interviews will be conducted within 3 months of completion of the training activity. It is estimated that seven respondents per grantee will be sufficient to ensure saturation of themes in the content analysis of results from the qualitative interviews.
Short Message Service Survey (SMSS). The target population is students enrolled in each Campus at years 1 and 3 of the grant funding. Each year, the list of mobile phone numbers for all students will be obtained from each campus. A random sample of mobile phone numbers will be selected. The target number of respondents will be 100 per campus. It is expected that 1,000 mobile phone numbers will be required to achieve 100 responses. The list of mobile phone numbers from year 3 will be compared to that of year 1 to identify a stratum of mobile phone numbers present both years and to determine its relative size. Respondents in year 1 will be contacted again in year 3 if their mobile phone number is still present in the year 3 list. Oversampling mobile phone numbers present in both years will result in a more precise estimate of change.
Power Estimation
The minimum detectable difference—with desired power (80%) and significance level (5%)—for a change in a proportion of interest (e.g., proportion of students who attempted suicide in the last year) between the two waves of administration of SMSS was estimated for a sample size of 2,500 students per wave (corresponding to a typical cohort of 25 campuses).
Method
For the present power analysis the null hypothesis is that there is no difference in the proportion of interest between the two waves of administration; the alternative hypothesis is that the two proportions differ in any direction. Though the design calls for some overlap between the samples from the two waves, a two-sample test is used to compare the proportions. This is a conservative approach consistent with no overlap (or no correlation between observations from the same student, given some overlap).
The assumed target of inference is a typical cohort of campus grantees1. On average, such a cohort includes 25 campuses, which results in a total sample size of 2,500 students per wave. The design calls for equal sample sizes per campus. However, campus sizes (in terms of number of students) are not homogeneous. This will result in unequal probability of selection across students from different campuses and a variance increase compared to simple random sampling. This impact may be quantified by the Design Effect (DEFF). Based on typical variation in campus sizes, a DEFF close to three is anticipated.
Results
The table below presents the minimum detectable difference with the desired power and significance level for a set of proportions with different initial level. In sum, a sample of 2,500 students per wave has 80% power to detect at a 5% significance level, a difference between 4.5% and 6.8%, depending on the initial proportion. Along the result for the overall sample, results are shown for domains representing 50%, 25% and 10% of the sample. In sum, in domains comprising 50% to 25% of the sample it would be possible to detect small changes (using Cohen’s conventional labels), while it will be only possible to detect medium size effects in small domains (i.e. comprising 10% of the sample). Changes of that magnitude are conceivable for the measures of exposure to GLS activities, but are not expected for youth suicidal behavior.
Table 10. SMSS: Minimum detectable difference (between waves of administration) with 80% power at 5% significance level
Initial proportion |
Overall |
Domain |
||
50% of the sample |
25% of the sample |
10% of the sample |
||
10% |
4.5% |
6.6% |
9.7% |
16.7% |
20% |
5.8% |
8.3% |
12.0% |
19.7% |
30% |
6.5% |
9.2% |
13.2% |
21.2% |
40% |
6.8% |
9.6% |
13.7% |
21.5% |
50% |
6.8% |
9.7% |
13.6% |
21.1% |
Student Awareness Intercept Survey (SAIS). Respondents for the SAIS will represent a sample of the student population at up to four selected campuses. Campuses implementing targeted suicide prevention campaigns will be identified and selected by reviewing grant applications and through technical assistance activities. A sampling plan to obtain 400 student respondents at up to four participating campuses will be developed by the cross-site evaluation team in conjunction with the campus project team using geographical and temporal sampling frames of student activity. Working with the campus grantee, the evaluation team will recruit respondents utilizing a systematic process that randomly selects campus locations and times. For the follow-up administration, the same sample size will be targeted. However, that sample will result from a combination of follow-up interviews with students from the initial sample, in combination with students newly recruited through an intercept procedure similar to the procedure.
Power Estimation
The minimum detectable difference—with desired power (80%) and significance level (5%)—in a set of selected variables between the two waves of administration of SAIS was estimated for a sample of 400 students per campus per wave. To increase the precision in the estimation of change, the sample was designed to obtain a substantial overlap between administrations. The gains in precision due to such a design depend not only on the extent of the overlap, but also on the correlation among repeated observations on the same individual, both difficult to determine a priori. This difficulty is overcome by providing estimations for two scenarios, complete and no overlap between samples.
Method
For the present power analysis, the null hypothesis is that there is no difference in the mean value of the variable of interest across two waves of administration; the alternative hypothesis is that the two means differ in any direction. Two estimations are presented. For continuous variables, the first estimation is based on a t-test of difference in means for two independent samples, which would be adequate if the samples for the two administrations did not overlap at all. The second estimation is based on a paired t-test, which would be adequate if the two samples overlapped completely. In addition, for the second test, the standard deviation for the pair-wise difference between administrations is assumed to be equivalent to the standard deviation of the measure in each wave. This is equivalent to assume a correlation of .5 across two measures from the same youth. Finally, for the discrete variable, the scenarios correspond to one and two sample test of proportions.
Results are presented for a standardized continues variable (i.e., with a mean of 0, a standard deviation of 1). In addition, include examples of two constructs that will be assessed by SAIS: (1) stigma toward seeking mental health treatment and (2) knowledge of myth and facts about suicide and its prevention are included. Similar constructs were assessed on a previous survey to campus students (SPEAKS) as part of the GLS national evaluation through simple mean scores based on a set of four 5-point Likert questions and 28 dichotomous questions. The estimated means and standard deviation are based on a sample of approximately 20,000 students from 53 different campuses from that survey. Finally, results are presented for a discrete variable assumed to be close to 50% in the first administration (the most demanding situation).
Results
The table below presents the minimum detectable difference for each variable for the desired power and significance level for the two scenarios, complete overlap and no overlap between samples. In sum, a sample of 400 students per campus in each wave has 80% power to detect at a 5% significance level a relatively small difference, between 15% and 20% of the standard deviation of the variable. Along the result for the overall sample, results are shown for domains representing 50%, 25% and 10% of the sample. In sum, in domains comprising 50% to 25% of the sample it would be possible to detect small effects (using Cohen’s conventional labels), while it will be only possible to detect medium size effects in small domains (i.e. comprising 10% of the sample). Increased precision can be obtained, however, by combining samples across the four participating campuses.
Table 11. SAIS: Standardized Minimum detectable difference (across waves of administration) with 80% power at 5% significance level
Estimated Values |
Overall |
Domain |
||
50% of the sample |
25% of the sample |
10% of the sample |
||
|
No Overlap- Complete Overlap |
No Overlap- Complete Overlap |
No Overlap- Complete Overlap |
No Overlap- Complete Overlap |
Perception of Seeking Mental Health Treatment (Mean=2.15; SD=0.54) |
0.107-0.076 |
.152-.108 |
.215-.153 |
.343-.245 |
Knowledge of Myths and Facts about Suicide (Mean=0.73; SD=0.11) |
0.022-0.015 |
.031-.022 |
.044-.031 |
.070-.050 |
Standardized Variable (Mean=0; SD=1) |
0.198-0.140 |
.281-.199 |
.398-.283 |
.634-.454 |
Proportion (Mean=0.5; SD=0.5) |
0.098-0.070 |
.140-.100 |
.199-.141 |
.317-.227 |
Data collection activities for the Early Identification, Referral and Follow-up Analysis (EIRF), Early Identification, Referral and Follow-up Screening Form (EIRF-S), MIS Data Collection Activity and Training Activity Summary Page (TASP) are utilizing existing data maintained in grantee reporting systems. Statistical methods are not applied for these activities. All the power analyses were based on the targeted sample size of actual respondents. For this reason, the power analyses inherently reflect sampling error. To achieve the targeted number of respondents, however, a larger sample of cases must be sampled due to different forms of non-response. Table 11 presents the estimated number of cases needed to achieve the desired sample size by instrument.
Table 12. Estimated Number of Cases Needed to Achieve Sample Size
Instrument Version |
Target sample size of respondents |
Total sample size |
Source |
State/Tribal TUP-S |
2,000 respondents per year |
5,000 consents to participate |
Based on TUP-S response rates among trainees who consent to be contacted of 40%; will be revised based on pilot |
State/Tribal TUP-S 6-Month Follow-up* |
600 respondents per year (pilot 200) |
1,200 TUP-S respondents (pilot 400) |
Conservative estimate of 50% lost to follow up; will be revised based on pilot. |
Campus TUP-S |
500 respondents per year (pilot 100) |
1,250 consents to participate |
Based on ST TUP-S response rates among trainees who consent to be contacted of 40%; will be revised based on pilot |
Adolescent TUP-S |
400 respondents per year (pilot 100) |
1,000 consents to participate (250 pilot) |
Conservative estimate based on typical phone survey response rates of 40%; will be revised based on pilot |
SMSS |
100 respondents per campus and administration |
1,000 cellphones per campus and administration |
Conservative estimate based on typical mail response rates of 10% |
SAIS |
400 respondents per campus and administration |
700 attempts to obtain consent at baseline; at follow up all respondents to baseline and 350 new attempts to obtain consent. |
Based on SPEAKS intercept administrations in two campuses where refusal rates were estimated at between 35-40%, and a conservative estimate of 50% lost to follow up. |
Prevention Strategies Inventory (PSI). Respondents for the Prevention Strategies Inventory will be project evaluators and/or program staff. Each of the 61 State/Tribal grantees and 60 Campus grantees will be required to complete the inventory. The baseline version of this Web-based inventory will be implemented following the first 2 quarters in year 1 of the grantee’s funding period and, thereafter, quarterly over the duration of the grant period. The cross-site evaluation team will provide a Web-based platform for data entry, will train program staff to complete the inventory, and will monitor completion. Each grantee will be provided via email a unique username and password to log in to the Web-based inventory. No individual identifying information will be provided when completing the inventory. Logging in and completing the inventory will imply consent for completion.
Training Exit Survey (TES) Individual Forms. All individuals involved in training activities at each of the 60 Campus grantee sites will be asked to complete the Training Exit Survey. Upon completion of a training activity, local program staff and/or project evaluator will be responsible for providing the Training Exit Survey to participants for self-administration and immediate return. The survey cover page introduces the survey and explains the consent process. Consent will be implied based on completion and submission of the survey to program and/or evaluation staff. A scannable survey option will be made available or, as an alternative, the survey can be administered in a paper-and-pencil format. If using the scannable surveys, local program staff will collect completed surveys and forward to the cross-site evaluation team. If paper-and-pencil surveys are used, local program staff will be responsible for entering survey data into the Web-based data collection system. Participation in the Training Exit Survey will be voluntary but a survey will be offered to all training participants.
Training Utilization and Preservation – Survey (TUP-S): State/Tribal, 6-Month Follow-up, and Campus Versions. The Training Utilization and Preservation – Survey (TUPS-S): State/Tribal Version will be administered to a random sample of 2,000 State/Tribal training participants per year. The Training Utilization and Preservation-Survey (TUP-S): Campus Version of the instrument will be piloted with 100 students the first year, and will be administered to 500 students in subsequent years. At the time of a training event at both State/Tribal and Campus grantee sites, respondents will be asked to complete a consent-to-contact form indicating their willingness to be contacted by the cross-site evaluation team to participate in the TUP-S. Participants from State/Tribal grantees will also consent to be recontacted in 6 months for participation in the Training Utilization and Preservation-Survey (TUP-S): State/Tribal 6 Month Follow Up. Consent-to-contact forms will be returned to local program staff. Local program staff will forward consent forms to the cross-site evaluation team. Because it will be necessary to facilitate administration of the interview, identifying information for each key informant will be forwarded to the cross-site evaluation team. The cross-site evaluation team will contact a random sample of key informants via telephone 3 months following the training activity to introduce the study, request participation, and to schedule an appointment for administration of the interview. ICF Macro telephone interviewers will be responsible for administering the interview and will be trained by the cross-site evaluation team to administer the survey. Prior to administration of the TUP-S, each respondent will provide verbal consent.
Training Utilization and Preservation–Survey (TUP-S): Adolescent Version. As part of a 1-year pilot, the Training Utilization and Preservation–Survey (TUP-S): Adolescent Version will be administered to 100 youths under the age of 18 who have participated in youth-targeted trainings sponsored by State/Tribal grantees. Grantees sponsoring trainings for youth will be identified during start-up and TA calls. Trainings will be selected based on whether a process to gather caregiver/youth consent is in place, and the grantee is able to incorporate the TUP-S consent-to-contact process into the general training consent procedures. Consent for a youth’s participation in a training activity will be collected from the youth’s caregiver/guardian and from the youth by the grantee program staff or training facilitator prior to the training. ICF Macro staff will work with grantees to collect parental and youth consent for the youth participation in the TUP-S. Contact information for the TUP-S will be collected via the consent-to-contact form that will be distributed with the consent for participation in the training event by the grantee and training coordinator. The consent-to-contact form will include a training ID and ask participants to provide identifying information (name, cell phone number, and mailing address) necessary for contacting them for the TUP-S and for administering the incentive.
Two methods to reach youth for the TUP-S will be piloted. The first is using a Web survey. Youth will be contacted using their preferred email from the consent-to-contact form. When consent-to-contact information is received form the grantee, the youth will receive a thank-you email and a reminder that he or she will receive a Web survey in 3 months. One week prior to the administration of the 3-month follow-up survey, the youth will receive another reminder email about the survey. Three months following the training event, youth participants will receive a link to a Web-based survey. The Web-based survey will be maintained on an online data management system: the Suicide Prevention Data Center (SPDC). The youth will be reminded and thanked for their consent to participate that was provided after the training event. After completing the survey, the youth will be given a choice of incentive (an online gift code or mailed VISA gift card). If the youth selects the VISA gift card, he or she will be asked to enter contact information where the card should be mailed.
The other method to be piloted is a Short Message Service Survey (SMSS), or text message survey. When consent-to-contact information is received form the grantee, the youth will receive a thank-you text and a reminder that he or she will receive a text survey in 3 months. One week prior to the administration of the 3-month follow-up survey, the youth will receive another reminder text about the survey. Three months following the training event, the youth will receive a text message to the cell phone number entered on the consent-to-contact form. The youth will be reminded of the consent to participate that he or she provided at the training event three months prior. The youth will then receive a series of five questions via text message, and will provide response via text. At the conclusion of the survey, respondents will receive an electronic gift code via text message.
Referral Network Survey (RNS). The Referral Network Survey (RNS) will be administered once in the second year of grant funding, and again in year 3 of grant funding to each of the 61State/Tribal grantees. Using zip code data submitted by grantees on the Training Activity Summary Page (TASP), cross-site evaluation staff will determine the county or region where the grantee has the greatest impact. The grantee will then be asked to provide contact information for at least one and up to three organizations in this country or region. Cross-site evaluation staff will make a preliminary phone call to ask these primary organizations for their referral network. Using snowball sampling to determine the entire referral network for the county or region, the cross-site evaluation staff will contact all organizations within the referral network to conduct the RNS. Snowball sampling will be repeated until saturation is reached.
If the participant agrees to participate in the survey during the initial phone calls, respondents will be asked to provide a current email address. Once the referral network has been established, respondents will be sent an online survey. This online survey will be prefilled with the entire list of the network so respondents may select which organizations are in their direct referral network. Respondents will have 2 weeks to complete the survey; if, they do not respond after 2 weeks, cross-site evaluation staff will remind individual respondents via a phone call. Prior to the administration of the survey in year 3, cross-site evaluation staff will call all of the agencies that participated in the survey in year 1 to confirm that the contact person is still present at the agency and also to see whether any new organizations have been included in their network. If applicable, these new organizations will be included in the second round of administration. The same data collection procedures will be used for the first and second administrations of the survey.
Coalition Survey (CS). The Coalition Survey (CS) will be administered to all State/Tribal grantees that indicate participation in coalition-building activities in their Prevention Strategies Inventory (PSI) once in the year 1 of the grant, and again during the year 3 of grant funding. Each grantee will be asked to provide the names and contact information of up to 10 individuals identified as part of the suicide prevention coalition. These individuals will be contacted initially by phone by the cross-site evaluation team, and then asked for an email address. Respondents will then be sent a link to the online survey. For the second administration of the survey in year 3, the grantee will be asked to confirm that the previous respondents are still participants in the coalition and to provide any updated contact information as necessary .The same data collection procedures will be used for the first and second administrations of the survey.
Coalition Profile (CP).The Coalition Profile will be administered once to all State/Tribal grantees who indicate their involvement with coalition-building efforts in their Prevention Strategies Inventory (PSI). These grantees will be contacted via email by the cross-site evaluation staff and asked to complete a short online survey about their primary coalition.
Short Message Service Survey (SMSS). The Short Message Service Survey (SMSS) will be administered to Campus grantees once during year 1 of grant funding, and again in year 3 of grant funding. In each of these years, the list of mobile phone numbers for all students at each of the 60 Campus grantee sites will be obtained by the cross-site evaluation team through one of three methodologies: directly from the campus, via grantee-led intercept/recruitment, or via ICF Macro intercept/recruitment. A random sample of mobile phone numbers will be selected to be contacted for survey participation. The target number of respondents will be 100 per campus. The list of mobile phone numbers in year 3 will be compared to that of year 1 to identify a stratum of mobile phone numbers present both years. Respondents from year 1 will be contacted again in year 3 if their mobile phone numbers are still present in the list for year 3.
Each respondent will receive an initial text message asking if they want to participate in a text survey about suicide prevention. Upon responding “yes,” the respondent will receive two to three text messages containing consent language, including information about the background, risks and benefits to the study, compensation, as well as information to contact the National Suicide Prevention Hotline if they require help. Respondents who choose to continue after the consent process will receive four questions total with one question per text message. Each text message will also contain response options, including an option to skip the question. Participants will be asked to reply to the text with their response. Subsequent questions will be sent upon receipt of response to the previous question. Respondents completing the survey or opting out will receive a thank- you text with contact information for the cross-site evaluation team and a reminder of the National Suicide Prevention Hotline number. The survey will remain available for 24 hours after the first message. There is the option of resending the initial message at a later date.
Student Awareness Intercept Survey (SAIS). Respondents to the Student Awareness Intercept Survey (SAIS) will represent a sample of the student population at up to four selected campuses . Campuses implementing targeted suicide prevention campaigns will be identified and selected by reviewing grant applications and through technical assistance activities. A sampling plan to obtain 400 student respondents at up to four participating campuses will be implemented by the cross-site evaluation team in conjunction with the campus project team using geographical and temporal sampling frames of student activity. Working with the campus grantee, the evaluation team will recruit respondents utilizing a systematic process that randomly selects campus locations and times. The baseline SAIS will be administered by ICF Macro staff onsite at participating campuses. ICF Macro will utilize screening questions to ensure participants are students of the campus prior to participation. Respondents will participate in the Web-based survey on tablets or laptop computers provided by ICF Macro staff.
After participation, respondents will be asked to provide their email address and phone number for participation in a 3-month follow-up survey. Participants for 3-month follow-up SAIS implementation will be recruited via a multimode method (4 emails and a final phone call). Students who participated in the baseline SAIS will be emailed a password and URL to complete the follow-up survey online.
Life Skills Activities Follow-up Interview (LAFI). The Life Skills Activities Follow-up Interview (LAFI) will be administered to respondents from Campus grantee sites 3 months following their participation in life skills and wellness training activities specifically for students on campus. The cross-site evaluation team, in consultation with local program staff, will select five particular training activities per year for which to administer the LAFI. Trainees will be asked to complete a consent-to-contact form indicating their willingness to be contacted to participate in the LAFI and return the form to local program staff. Key informants for the LAFI will be randomly selected from those individuals who consent to be contacted by the cross-site evaluation team. Local program staff will forward the contact consent forms to the cross-site evaluation team. Up to seven respondents from each of the five selected trainings will be randomly selected from among the potential respondents based on consent-to-contact information, for a total of up to 35 respondents per year. Interviews will be conducted within 3 months of completion of the training activity. The cross-site evaluation team will contact each identified key informant via telephone within 3 months of the training to introduce the study, request participation, and to schedule an appointment for administering the interview. Prior to administration of the LAFI, each respondent will provide verbal consent. The interviews consist of a series of open-ended questions. Participants will be contacted via telephone and asked the questions in the LAFI guide. Interviews will be audio recorded with the respondents’ permission and transcribed for analysis.
Participation in the cross-site evaluation is a requirement of the GLS Suicide Prevention Program. Therefore, completion of the Prevention Strategies Inventory by program staff will be a requirement. However, the cross-site evaluation team has taken a number of steps to minimize the burden on local programs to ensure that completion is timely. These steps include developing a Web-based data collection system, and providing training and TA to each grantee.
The cross-site evaluation team also will provide TA and training to all grantee sites to maximize response rates for the other data collection activities. This will be done by providing Web cast trainings, distributing data collection procedures manuals, conducting onsite training visits for the State/Tribal grantees, and providing ongoing one-on-one contact with each grantee through a TA liaison.
Methods that will be used to maximize response rates for the Referral Network Survey and the Coalition Survey include utilizing a Web-based data collection system for administration of the instrument, and providing training and TA for completing the survey. Local program staff will also be utilized to obtain contact information for respondents, and respondents will be contacted via telephone to remind them to complete the Web-based survey.
To increase response rates to the Student Awareness Intercept Survey (SAIS), the baseline version of the survey will be administered onsite at up to four campuses by cross-site evaluation staff. A Web-based version of the survey will also be utilized for easy implementation. Based on our experience implementing a similar survey with students, participants for the 3-month follow-up SAIS implementation will be recruited and reminded of the survey via a multimode method (4 emails and a final phone call). Response rates to the Short Message Service Survey will be augmented by conducting the survey via text message, allowing students to complete the survey at a time convenient for them and via technology with which they are familiar. The Short Message Survey, which replaces the Suicide Prevention, Exposure, Awareness and Knowledge Survey, has a significant reduction in the number of questions in an effort to increase response rates.
Methods that will be used to maximize response rates for the qualitative interviews (i.e., the Training Utilization Preservation Surveys: State/Tribal, Campus and Adolescent Versions and Life Skills Activity Follow-up Interviews) include obtaining buy-in from key program stakeholders, providing flexibility in scheduling, and conducting follow-up phone calls and emails to nonresponders. In addition, local program staff will be utilized to obtain contact information for respondents, which will result in more accurate information, thus increasing response rates. If any identified respondents for the qualitative interviews are nonresponsive, the cross-site evaluation team will request that local program staff identify replacement respondents.
Table 12 provides details on the response rate and psychometric analyses for the instruments that have been collected to date and their relevance to the revisions to the new protocols.
Table 13. Previously Implemented Instruments and Revisions
Instrument |
Grantee Program |
Newly Proposed Instrument? |
Response Rate & Psychometric Analyses Information2 |
Revisions to Proposed Protocol 3 |
Prevention Strategies Inventory (PSI)
|
Campus & State/ Tribal |
Revised |
Response Rate: Almost all grantees participate in the PSI, among currently funded grantees 100% of campus grantees participated in the last administration of the PSI and over 95% of the currently funded States and Tribes participated
Psychometric Analyses: Not appropriate for the PSI. |
Featured Changes: PSI will gather more information about the populations strategies and about wellness and coalition activities.
No revisions related to response rates proposed; content changes only.
|
Training Exit Survey (TES) Cover Page & TES Individual forms (Core, Gatekeeper, Clinical and Campus Connect Versions)
|
Campus & State/ Tribal |
Revised |
Response Rate: TES participation among currently funded Campus grantees has ranged from 64.1% - 95.2%. The TES cover page can be used to reconcile grantee individual-level and aggregate training data.
Psychometric Analyses: Not appropriate for the TES individual form versions or the TES Cover Page/TASP. |
Featured Changes: The TES individual form was discontinued among State and Tribal grantees and new Campus grantees.
The TES cover page was renamed the Training Activity Summary Page (TASP), and collects information about intended outcomes, and booster and follow-up trainings.
No revisions related to response rates proposed; content changes only.
|
Training Utilization & Preservation Survey (TUP-S) |
Campus & State/ Tribal |
Revised |
Response Rate: Survey response rates were 25% in FY 2012.
Psychometric Analyses: Not appropriate for the TUP-S Versions. |
Featured Changes: A participant sign-in sheet has been introduced for the TUP-S to help increase the number of consent-to-contact forms from grantees. Call center procedures related to varying the times of day and the contact number utilized to reach respondents has been modified to improve survey response rates.
New Instrument(s): The TUP-S State/Tribal has been expanded to include a 6 month survey. A TUP-S Campus version has also been introduced to assess campus trainee behaviors, and an adolescent version of the TUP-S will be piloted to gather information from youth 12-17 using text-message technology for State/Tribal grantees. |
Referral Network Survey (RNS) |
State/ Tribal |
Revised |
Response Rate: Previous administrations of the RNS with cohorts 1-6 have yielded between 44-81% agency response rates.
Psychometric Analyses: Not appropriate for the RNS, CP or CS. . |
Featured Changes: The RNS will be implemented as a web-based survey and use snowball sampling to identify grantee networks to include organizations more closely related to the grantee referral network and improve response rates among all grantees.
New Instrument(s): A Coalition Profile and Survey have been introduced to gather information about coalitions that are critical to the success of grantee programs. |
Suicide Prevention Exposure, Awareness, and Knowledge Survey (SPEAKS)*
|
Campus |
Discontinued |
Response Rate: RR from the last administration of the SPEAKS ranged from 4.1 – 76.1% - with an average rate of 21.5%.
Psychometric Analyses: Psychometric analyses are applicable to some data elements collected as part of the SPEAKS. Psychometric analyses may be possible with data collected via the SAIS. |
Featured Changes: The SPEAKS has been discontinued and replaced by the SMSS and the SAIS.
New Instrument(s): The Short Message Service Survey (SMSS) SMSS is a brief text-message survey conducted to assess exposure to campus suicide prevention activities and personal history of suicide attempts and ideation.
The Student Awareness Intercept Survey (SAIS), a Web-based student survey with baseline and follow-up versions, will be conducted as a case study of the reach and impact of campus social marketing campaigns related to suicide prevention. The SAIS retains some of the subscales previously used in the SPEAKS. |
Early Identification, Referral, and Follow-up Individual and Aggregate Form (EIRF) |
State/ Tribal |
Revised |
Response Rate: While response rates are not available, we monitor the participation of grantees in each activity. Over 80% (81.6%) of cohort 6 grantees are participating in the EIRF, 13.0% of cohort 7 grantees are participating in the EIRF. The EIRF doesn’t have an identified sample and therefore response rate information is not applicable.
Psychometric Analyses: Not appropriate for the EIRF |
Featured Changes: The EIRF will continue to collect individual-level information about youth identified at-risk by gatekeepers and/or via screening tools--Initial referral follow-up information should be obtained within 3 months and details from the second follow-up should be gathered within 6 months. The EIRF-S now collects aggregate-level information on youth identified as at-risk via screening tools used by GLS grantee. No revisions related to response rates proposed; content changes only.
|
Management Information System Data Abstraction & Submission (MIS activity) |
Campus |
Revised |
Response rate: Almost all grantees participate in the MIS, among currently funded grantees 100% of campuses participated in the last MIS administration.
Psychometric Analyses: Not appropriate for the MIS. |
Featured Changes: The MIS will request information about the number of student suicide attempts and completions for 5 years previous to grant funding at baseline and for the previous Academic year thereafter.
No revisions related to response rates proposed; content changes only.
|
The GLS Suicide Prevention and Early Intervention Program is the first federally funded program to support suicide prevention programs in State/Tribal communities and Campuses. Drawing upon our experience of 3 years of data collection for the cross-site evaluation and feedback from grantees, improvements have been made to the administration protocols and content of cross-site evaluation data collection instruments.
As new measures were developed, standard instrument development procedures, including review of the literature, item development, and content review by experts in the field were used. All instruments underwent cognitive and/or pilot testing, and/or expert review. These procedures were used to enhance question accuracy and determine administration times. In addition, Web-enabled and SMS instruments will undergo usability testing prior to fielding. Usability testing refers to pilot testing of the interface for administering questionnaires to determine the most efficient and understandable presentation. Typically, this is completed with a prototype and modifications are made before final fielding.
First, a thorough review of the literature was conducted related to suicide prevention training activities and suicide awareness and knowledge in efforts to develop the Training Exit Survey, the Training Utilization and Preservation Survey, and the Student Awareness Intercept Survey. In addition, experts in mental health referral networks were consulted in developing the Referral Network Survey. Second, drafts of the instruments were developed and reviewed by cross-site evaluation team members, survey methodologists, representatives from SAMHSA, and content experts in the field of suicide prevention. Third, item analyses were conducted across instruments to be sure that key critical items were assessed similarly across all questionnaires. Fourth, in an effort to enhance question accuracy and determine administration time, the revised instruments underwent cognitive testing and/or pilot testing on no more than nine respondents matching the type appropriate for the instrument.
The cross-site evaluator has full responsibility for the development of the overall statistical design, and assumes oversight responsibility for data collection and analysis. Training, TA, and monitoring of data collection will be provided by the cross-site evaluator. The individuals responsible for overseeing data collection and analysis are:
Christine M. Walrath, PhD
ICF Macro, Inc.
116 John Street, Fl. 8
New York, NY 10038
(212) 941-5555
The following individuals will serve as statistical consultants to this project:
Christine M. Walrath, PhD
ICF Macro, Inc.
40 Wall Street, 34th Floor
New York, NY 10005
(212) 941-5555
Robert Stephens, PhD
ICF Macro, Inc.
3 Corporate Square, Suite 370
Atlanta, GA 30329
(404) 321-3211
Ye Xu, MS
ICF Macro, Inc.
3 Corporate Square, Suite 370
Atlanta, GA 30329
(404) 321-3211
Lucas Godoy Garraza, MA
ICF Macro, Inc.
40 Wall Street, 34th Floor
New York, NY 10005
(212) 941-5555
David Goldston, PhD
Duke University
Duke Child and Family Study Center
718 Rutherford Street DUMC 3527
Durham, NC 27710
(919) 416-2423
The agency staff person responsible for receiving and approving contract deliverables is:
Melanie Brown, MPH, MA
Suicide Prevention Branch
Center for Mental Health Services
Substance Abuse and Mental Health Services
1 Choke Cherry Road
Room 6-1105
Rockville, MD 20857
Phone: (240) 276-1873
American Association of Suicidology (2010). Youth Suicidal Fact Sheet: Based on 2010 Data. Retrieved February 2010, from
http://www.suicidology.org/c/document_library/get_file?folderId=262&name=DLFE-627.pdf
American College Health Association. (2012, Spring). American College Health Association-National College Health Assessment II : Undergraduate Reference Group Executive Summary Hanover, MD.
Cash, S.J., Bridge, J.A. (2009, October). Epidemiology of Youth Suicide and Suicidal Behavior. Current Opinion in Pediatrics, 21(5), 613-619.
Dillman, D. (2000). Mail and Internet Surveys-Second Edition. New York, NY: John Wiley & Sons, Inc.
Eggert, L.L., Nicholas, L.J., Owen, L.M. (1995). Reconnecting Youth: A Peer Group Approach to Building Life Skills. National Educational Service, Bloomington, IN.
Eggert, L.L., Randell, B.R., Thompson, E.A., Johnson, C.L. (1997). Washington State Youth Suicide Prevention Program: Report of Activities. University of Washington, Seattle, WA.
Institute of Medicine (2002). Goldsmith, SK, Pellmar TC, Kleinman, AM, Bunney, WE, Editors, Committee on Pathophysiology & Prevention of Adolescent & Adult Suicide; Board on Neuroscience and Behavioral Health. Reducing Suicide: A National Imperative. Retrieved February 2013, from http://www.nap.edu/catalog.php?record_id=10398
Kalafat, J., and Elias, M. (1994). An evaluation of a school-based suicide awareness intervention. Suicide and Life-Threatening Behavior, 24(3), 224-233.
King, K.A., Smith J. (2000). Project SOAR: A training program to increase school counselors’ knowledge and confidence regarding suicide prevention and intervention. Journal of School Health, 70(10), 402-407.
National Adolescent and Young Adult Health Information Center (2011). Data Profiles: National Summary. Retrieved February 2013, from http://nahic.ucsf.edu/resources-tools/national-summary/
O’Connor, R.C., Platt, S., Gordon, J. (2011). Twenty-Nine School-Based Strategies for Youth Suicide Prevention. International Handbook of Suicide Prevention: Research, Policy and Practice. Published online.
The Surgeon General’s Call to Action to Prevent Suicide (1999). Washington, DC: U.S. Public Health Service.
2012 National Strategy for Suicide Prevention: Goals and Objectives for Action (2012, September). Washington, DC: U.S. Department of Health and Human Services (HHS) Office of the Surgeon General and National Action Alliance for Suicide Prevention 2012.
1 Alternatively, a particular set of campuses in a cohort can be interpreted as a sample of a much larger set of campuses. In that case, each campus is a cluster (instead of a stratum), and the extent to which students from the same campus are similar must be taken into account. In both cases, there is an impact on the precision that is usually summarized by the DEFF.
2 Psychometric analyses are indicated for data collection instruments that apply scales that are able to be analyzed. If the activity doesn’t collect scaled data it is not considered appropriate for psychometric analysis and is thereafter indicated as “not appropriate”.
3 The approaches to address response rates and implementation challenges through protocol revision have been included in the last column of the Table 12.
Suicide
Cross-Site Evaluation Page
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-28 |