Supporting Statement B

Project AWARE-TISS Cross-Site Evaluation

Project AWARE and TISS Programs

Supporting Statement

2. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

Exhibit 1 displays the expected number of respondents to each data collection activity in which primary data is collected across the three-year OMB period.

Exhibit 1. Total Number of Respondents by “Primary” Data Collection Activity

Instrument	Participating Grantees* per Year	Respondents per Grantee per Year	Total Respondents per Year
IS	47	Approximately 4 AWARE Approximately 2 TISS	282
IKII	47	Approximately 6	282
YFFG-Y**	8	Approximately 10	80
YFFG-F**	8	Approximately 10	80
CPS	47	Approximately 6 AWARE Approximately 3 TISS	423
TSF	47	Approximately 10	470
PFF	47	Approximately 75 AWARE Approximately 25 TISS	2,775
APPTS	32	Approximately 125 AWARE	4,000
TPPTS	15	Approximately 50 TISS	750
WFS	47	Approximately 63 AWARE Approximately 25 TISS	2,391
PCSS	47	Approximately 6	282
STCSS	47	Approximately 6	282
SSCSS	47	Approximately 10	470
SIRF	47	Approximately 100	4,700

*There are 32 AWARE grantees and 15 TISS grantees, totaling 47 participating grantees.

**For the Youth and Family Focus Groups (YFFG-Y or YFFG-F), approximately 16-18 focus groups will be conducted annually such that all grantees will participate in a focus group at least once during the 3-year OMB period.

Nonprobability (purposive) sampling will be used for the following activities: Implementation Survey (IS), Implementation Key Informant Interviews (IKII), Youth and Family Focus Groups (YFFG-Y, YFFG-F), Collaboration and Partnership Survey (CPS), Participant Feedback Form (PFF), AWARE and TISS Pre-Post Training Surveys (APPTS and TPPTS),Workforce Follow-up Survey (WFS), and Climate and Safety Surveys for parents (PCSS), students (STCSS) and school staff (SSCSS).The following data collection activities are reports on grant activities or existing data abstractions required from every grantee, so no sampling is required: Training Summary Form (TSF) and Student Identification and Referral Form (SIRF). Respondents to these activities will be grantee program staff.

For the APPTS and TPPTS, a power analysis was conducted to determine the minimal detectable effect size for change in training participants’ outcomes pre- to post-, per grantee site, per year. Recruitment of the sample will continue until a total of 125 trainees per AWARE grantee and 50 trainees per TISS grantee per year participate in the pre-post surveys. These sample sizes will be sufficient to detect the effect size of f = 0.25 (per AWARE grantee) and f = 0.40 (per TISS grantee).

2. Information Collection Procedures

Information collection procedures for the Project AWARE-TISS Cross-Site Evaluation instruments are described in Exhibit 2.

Exhibit 2. Information Collection Procedures

Instrument	Procedures
IS	Once per year, grantee staff will be asked to confirm a list of names and email addresses for project coordinators for the AWARE and/or TISS program as well as representatives of agencies or organizations with which they partner for the TISS program. A purposive sample of four AWARE project coordinators per AWARE grantee and two identified TISS partner/agency representatives per TISS grantee that have already agreed to participate will be contacted annually by email and provided a web link to the electronic consent form; if participants consent, they will be directed to complete the web-based survey hosted on the AWARE-TISS Online Data System (ATODS).
IKII	Once per year, grantee staff will be asked to confirm a list of names and email addresses for project coordinators for the AWARE and/or TISS programs as well as representatives of agencies or organizations with which they partner. A purposive sample of six individuals per grantee inclusive of project coordinators and identified partner/agency representatives will be contacted by email and invited to participate in the IKII. Technical assistance liaisons will then reach out to potential respondents who have agreed to participate via email and phone to schedule the interviews, to be conducted in person during site visits or remotely if needed. The evaluation team will provide an informed consent form via email in advance of the interview and obtain verbal consent immediately prior to each interview. Each interview will last approximately 45-60 minutes. Interviewers will review the consent form and answer any questions before beginning the interview. With the participant’s permission, interviews will be recorded and transcribed to support fact checking and to maintain the reliability and validity of the data.
YFFG-Y YFFG-F	Once per year, grantee staff will coordinate with partner agencies and organizations to recruit youth and/or family members of youth who have received services to participate in a focus group designed to improve understanding of youth AWARE or TISS program experiences. A combination of YFFG-Y and YFFG-F will be conducted onsite for up to 12 AWARE grantees per year and six TISS grantees per year such that all grantees have focus groups at least once during the three-year data collection period. Focus groups will include a purposive sample of up to 10 youth aged 14-18 (or older if appropriate) or 10 parents/family members of youth. The focus group is estimated to last 60-90 minutes. Focus groups will only be conducted with parental consent and youth assent after review of study information. Focus group moderators will review the consent form and answer any questions before beginning the focus group. With participants’ permission, focus groups will be recorded and transcribed to support fact checking and to maintain the reliability and validity of the data.
CPS	One per year, grantee staff will provide a list of names and email addresses for representatives of agencies or organizations with which they partner for AWARE and/or TISS. Then a purposive sample of six individuals per AWARE grantee and three individuals per TISS grantee (e.g., grantee staff and identified partner representatives who have agreed to participate) will be contacted by email and provided a web link to the electronic consent form and if they consent, will be directed to complete the web-based survey in ATODS.
TSF	Grantee staff responsible for training events will be asked to complete a brief TSF form in ATODS at least one month in advance of training events, to allow questions on the PFF to be tailored based on the training objectives. After the training is completed, grantee staff will be asked to update the training form if the length of training or number of participants was different than planned and reported in the TSF when submitted prior to the training.
APPTS TPPTS PFF WFS	During training events, all participants will be asked to complete either the APPTS or TPPTS. The surveys will be administered electronically (via URL or QR code at the time of the training). If a trainer is unable to administer the survey form electronically, or a trainee does not have access to a mobile device or computer, they may also complete the surveys on paper. The grantee will submit this information to ICF, through direct data entry into the ATODS, within two weeks of the training event. The APPTS and TPPTS include a consent to contact question indicating their willingness to be contacted to participate in subsequent surveys (the PFF and WFS). Once consent-to-contact has been received, participants who consent to participate in subsequent surveys will receive a link to the PFF on the device where they completed the post-survey and sent to the email address they provided. Following the training, ICF will contact a random sample of respondents for the WFS at three-months and 12-months following the training (approximately 50% of those who have completed the APPTS or TPPTS).
PCSS STCSS SSCSS	In Year One and Year Three of the evaluation data collection period, grantee staff will coordinate with partner local education agencies (LEAs) to provide a list of names and email addresses of six high school students and six parents (per grantee) from an identified priority school who are interested in participating. Grantee staff will also be provided with support in developing a flyer with information about the PCSS and STCSS to share with the identified parents to inform them about the surveys before they receive the email. Parents that opt to participate in the PCSS and are willing to consent for their children to participate will receive an electronic web link to access the electronic consent form and survey through ATODS. Students aged 14-17 whose parents provide consent will be sent an email with a link to an online assent form; those who assent will proceed to the survey. Students 18 and older will be sent an email with a link to an online consent form; those who consent will proceed to the survey. Also, in Year One and Year Three, grantee staff will coordinate with partner LEAs to provide a list of names and email addresses of 10 school staff (e.g., teachers and administrators) in the priority school identified who have agreed to complete a survey. Grantee staff will also be provided with support in developing a flyer with information about the SSCSS to share with the identified teachers and school staff to inform them about the surveys before they receive the survey web link via email. An email with a web link to access the consent and survey through ATODS.
SIRF	Grantee staff enter SIRF data via an online survey function within ATODS. Identification, referral, and follow-up information is entered during the same month as the student is identified as in need of additional behavioral health or trauma-related support services. Data are extracted from case records or other existing data sources, including any school staff, school-based mental health providers, or community/family members who make an identification. Grantees who do not have access to an existing tracking system should work with their technical assistance liaison and SAMHSA GPO to discuss approaches for adequately tracking and monitoring students identified and referred for services. Grantee staff will enter SIRF data into ATODS for up to 100 students annually throughout the 3-year data collection period.

3. Methods to Maximize Response Rates

Participation in the Project AWARE-TISS Cross-Site Evaluation is a requirement of the AWARE and TISS Programs. Therefore, completion of the TSF and SIRF by program staff will be a requirement. However, the evaluation team has taken several steps to minimize the burden on local programs to ensure that completion is timely. These steps include developing a web-based data collection system, using updated technology, and providing training and technical assistance to grantees. The evaluation team also will provide training and technical assistance to maximize response rates for the other data collection activities by hosting web trainings, and distributing procedural manuals, Specific methods to increase the response rates are provided in Exhibit 3 below.

Exhibit 3. Methods to Maximize Response Rates

Instrument	Methods to Maximize Response Rates
IS	The IS link will be generated within the ATODS or emailed to the respondent directly. Respondents will be encouraged to complete the survey. If a respondent starts the survey but does not complete it, they will receive up to two reminders via email or text message to encourage participation. A $20 gift card is provided to all respondents who complete the survey.
IKII	Evaluation team members will provide grantees with evaluation training and technical assistance (TTA), equipping them with the tools needed to identify potential interview respondents. Interview respondents will already be familiar with the grantee programs. A $25 gift card is provided to all respondents who participate in an individual interview.
YFFG-Y YFFG-F	Evaluation team members will provide grantees with evaluation TTA, equipping them with the tools needed to identify potential focus group respondents, including youth and parents/families. Focus group respondents will already be familiar with the grantee programs and receive services under the AWARE/TISS grants. A $25 gift card is provided to all respondents who participate in a focus group.
CPS	The CPS link will be generated within the ATODS or emailed to the respondent directly. Respondents will be encouraged to complete the survey. If a respondent starts the survey but does not complete it, they will receive up to two reminders via email or text message to encourage participation. A $20 gift card is provided to all respondents who complete the survey.
APPTS TPPTS	The APPTS and TPPTS links (one for pre- and one for post-) will be generated within the ATODS or emailed to an alternative contact1. Before and after the training, the trainer will distribute this link to all participants who attend the training. This link may be shared via email, a QR code, or text message. Participants will be encouraged to complete both surveys. A $20 gift card is provided to all respondents who complete the both the pre- and post- survey.
PFF	The PFF link will be generated within the ATODS or emailed to the respondent or an alternative contact2. Following completion of the APPTS or TPPTS, respondents who indicate their consent to be contacted for an additional survey will be directed to a link to the PFF on their device and the same link will be sent to them by email and/or text message. Participants will be encouraged to complete the survey following training. If a participant starts the feedback form but does not complete it, they will receive up to two reminders via email or text message to encourage completion. A $10 gift card is provided to all respondents who complete the feedback form.
WFS	The WFS link will be generated within the ATODS, emailed to the respondent, or to an alternative contact3 to share with trainees who consented to be contacted three- and 12-months post training. This link may be shared via email, a QR code, or text message. Participants will be encouraged to complete the surveys. If a participant starts the survey but does not complete it, they will receive up to two reminders via email or text message to encourage participation. A $20 gift card is provided to all respondents who complete the follow-up survey (for a total of $40 if the respondent completes both the three- and 12-month follow-up survey.)
PCSS STCSS SSCSS	Evaluation team members will provide grantees with evaluation TTA, equipping them with the tools needed to identify/notify potential respondents, including parents, youth, and school staff from partner LEA schools. The links for the PCSS, STCSS, and SSCSS will be generated within the ATODS, emailed to respondents directly, or emailed to an alternative contact4 to share with respondents (and their parent or guardian, if a youth) who agreed to participate and obtained parental consent (for youth). This link may be shared via email, a QR code, or text message. Respondents will be encouraged to complete the survey. If a respondent starts the survey but does not complete it, they will receive up to two reminders via email or text message to encourage participation. A $20 gift card is provided to all respondents who complete the survey.

4. Tests of Procedures

As new measures were developed, standard instrument development procedures, including review of the literature, item development, and content review by experts in the field were used. All instruments underwent procedures to enhance question accuracy and determine administration times. In addition, web-enabled instruments will undergo usability testing prior to fielding. Usability testing refers to pilot testing of the interface for administering questionnaires to determine the most efficient and understandable presentation. Typically, this is completed with a prototype and modifications are made before final fielding.

Drafts of the instruments were developed and reviewed by evaluation team members, survey methodologists, representatives from SAMHSA, and content experts in the field of behavioral health. Item analyses were conducted across instruments to be sure that key critical items were assessed similarly across all questionnaires. To enhance question accuracy and determine administration time, instruments underwent cognitive and/or pilot testing or expert review.

5. Statistical Consultants

ICF has full responsibility for the development of the overall statistical design and assumes oversight responsibility for data collection and analysis. Training, technical assistance, and monitoring of data collection will be provided by the Project AWARE-TISS Cross-Site Evaluation team. The individuals responsible for overseeing data collection and analysis are:

Megan Brooks, MA, PMP

ICF

Remote Office

Saint Paul, MN 55101

Phone: (612)253-9975

Email: [email protected]

Colleen Murray, DrPH, MPH

ICF

2635 Century Parkway Suite 1000

Atlanta GA 30345

Phone: (404)592-2286

Email: [email protected]

Anna Sheremenko, PhD, MS

ICF

2635 Century Parkway Suite 1000

Atlanta GA 30345

Phone: (706)201-7970

Email: [email protected]

India Rose, PhD, MPH, CHES

ICF

2635 Century Parkway Suite 1000

Atlanta GA 30345

Phone: (404)592-2232

Email: [email protected]

Elizabeth Douglas, MS

ICF

2635 Century Parkway Suite 1000

Atlanta GA 30345

Phone: (404)592-2175

Email: [email protected]

Jennifer Duffy, PhD

ICF

Remote Office

Columbia, SC 29044

Phone: (803)600-8550

Email: [email protected]

Rachel Ladd, PhD

ICF

Remote Office

Nashville, TN 37011

Phone: (865)806-5263

Email: [email protected]

Sarah Conklin, PhD, MS

ICF

Remote Office

Richmond, VA 23173

Phone: (540)420-9858

Email: [email protected]

The following individuals will serve as statistical consultants to this project:

Bob Stephens, PhD, MPH

ICF

2635 Century Parkway Suite 1000

Atlanta GA 30345

Phone: (404)320-4494

Email: [email protected]

Oliver Massey, PhD

University of South Florida

13301 Bruce Downs Blvd. MHC2-401

Tampa, FL 33612

Phone: (813)974-6403

Email: [email protected]

The agency staff person responsible for receiving and approving contract deliverables is:

Nainan Thomas, MA, LLB, MSW, PhD, LICSW

Senior Public Health Analyst

SAMHSA/CMHS/DPTSSP

5600 Fishers Lane

Rockville, MD 20857

Phone: (240)-276-1744

Email: [email protected]

References

Abadie, A., & Gardeazabal, J. (2003). The economic costs of conflict: a case study of the Basque country. American Economic Review, 93(1), 113–132. https://doi.org/10.1257/000282803321455188  

Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic control methods for comparative case studies: estimating the effect of California’s tobacco control program. Journal of the American Statistical Association, 105(490), 493–505. https://doi.org/10.1198/jasa.2009.ap08746

Abadie, A., Diamond, A., & Hainmueller, J. (2015). Comparative politics and the synthetic control method: comparative politics and the synthetic control method. American Journal of Political Science, 59(2), 495–510. https://doi.org/10.1111/ajps.12116  

Adler, K., Salanterä, S., & Zumstein-Shaha, M. (2019). Focus group interviews in child, youth, and parent research: an integrative literature review. International Journal of Qualitative Methods, 18. https://doi.org/10.1177/1609406919887274

American Academy of Pediatrics (2021). AAP-AACAP-CHA declaration of a national emergency in child and adolescent mental health. A declaration from the American Academy of Pediatrics. American Academy of Child and Adolescent Psychiatry and Children’s Hospital Association.

American Psychiatric Association. (2021). New APA poll shows sustained anxiety among Americans; more than half of parents are concerned about the mental well-being of their children. https://www.psychiatry.org/news-room/news-releases/new-apa-poll-shows-sustained-anxiety-among-america

Anthony, B., Banh, M. K., & Goldman, S. (2015). Mental health beliefs and literacy scale. Washington, DC: Georgetown University Child Development Center, National Technical Assistance Center for Children’s Mental Health.

Baffsky, R., Ivers, R., Cullen, P., Wang, J., McGillivray, L., & Torok, M. (2023). Strategies for enhancing the implementation of universal mental health prevention programs in schools: a systematic review. Prevention Science, 24(2), 337-352. DOI: 10.1007/s11121-022-01434-9

Bauer, C., Wakefield, J., Rue, H., Self, S., Feng, Z., & Wang, Y. (2016). Bayesian penalized spline models for the analysis of spatio-temporal count data. Statistics in Medicine, 35, 1848–1865. DOI: 10.1002/sim.6785

Bilbrey, J. B., Castanon, K. L., Copeland, R. B., Evanshen, P. A., & Trivette, C. M. (2022). Primary early childhood educators’ perspectives of trauma‑informed knowledge, confidence, and training. The Australian Educational Researcher. https://doi.org/10.1007/s13384-022-00582-9

Bitsko, R. H., Claussen, A. H., Lichstein, J., Black, L. I., Everett Jones, S., Danielson, M. L., et al. (2022, Suppl). Mental health surveillance among children—United States, 2013–2019. Morbidity and Mortality Weekly Report, 71(2), 1–42.

Breen N., Berrigan D., Jackson J. S., Wong, D. W. S., Wood, F. B., Denny, J. C., Zhang, X., & Bourne, P. E (2019). Translational health disparities research in a data-rich world. Health Equity 3(1), 588–600, DOI: 10.1089/heq.2019.0042

Brown, D. W., Anda, R. F., Tiemeier, H., Felitti, V. J., Edwards, V. J., Croft, J. B., et al. (2009, Nov). Adverse childhood experiences and the risk of premature mortality. American Journal of Preventive Medicine, 37(5), 389–396. 10.1016/j.amepre.2009.06.021

Byrd, C. M. (2017). The complexity of school racial climate: Reliability and validity of a new measure for secondary students. British Journal of Educational Psychology, 87, 700-721.

California Department of Education. (n.d.) California Healthy Kids Survey: A comprehensive student data collection system that addresses school climate, health risks and behaviors, and youth resiliency. https://www.cde.ca.gov/ls/he/at/chks.asp

California Department of Education. (n. d.) California School Staff Survey. https://www.cde.ca.gov/ls/he/at/cscs.asp

Castro, F. G., Kellison, J. G., Boyd, S. J., & Kopak, A. (2010). A methodology for conducting integrative mixed methods research and data analyses. Journal of Mixed Methods Research, 4(4), 342-360. DOI: 10.1177/1558689810382916. PMID: 22167325; PMCID: PMC3235529.

Centers for Disease Control and Prevention. (2021). Adverse Childhood Experiences Prevention Strategy. Atlanta, GA: National Center for Injury Prevention and Control, Centers for Disease Control and Prevention.

Centers for Disease Control and Prevention. (2023). WISQARS Injury data. WISQARS: web-based injury statistics query and reporting system. Division of Injury Prevention and Control. https://www.cdc.gov/injury/wisqars/index.html

Chapman, D. P., Whitfield, C. L., Felitti, V. J., Dube, S. R., Edwards, V. J., & Anda, R. F. (2004). Adverse childhood experiences and the risk of depressive disorders in adulthood. Journal of Affective Disorders, 82(2), 217–225. 10.1016/j.jad.2003.12.013

Dube, S. R., Felitti, V. J., Dong, M., Chapman, D. P., Giles, W. H., & Anda, R. F. (2003). Childhood abuse, neglect, and household dysfunction and the risk of illicit drug use: The adverse childhood experiences study. Pediatrics, 111(3), 564–572. https://doi.org/10.1542/peds.111.3.564

FAIR Health. (2021). The impact of COVID-19 on pediatric mental health: A study of private healthcare claims [White paper]. FAIR Health, Inc.

Felitti, V. J. (2002). The relation between adverse childhood experiences and adult health: Turning gold into lead. The Permanente Journal, 6(1), 44–47.

Felitti, V. J., Anda, R. F., Nordenberg, D., Williamson, D. F., Spitz, A. M., Edwards, V., et al. (1998). Relationship of childhood abuse and household dysfunction to many of the leading causes of death in adults. The Adverse Childhood Experiences (ACE) Study. American Journal of Preventive Medicine, 14(4), 245–258. https://doi.org/10.1016/s0749-3797(98)00017-8

Forman, S. G., Olin, S. S., Hoagwood, K. E., Crowe, M., & Saka, N. (2009). Evidence-based interventions in schools: Developers’ views of implementation barriers and facilitators. School Mental Health, 1(1), 26–36.

Freeman, Clarkson, P. A. (2014). Prevalence and relationship between adverse childhood experiences and child behavior among children. Infant Mental Health Journal, 35(6), 544–545.

Garraza, L. G., Walrath, C., Goldston, D. B., Reid, H., & McKeon, R. (2015). Effect of the Garrett Lee Smith memorial suicide prevention program on suicide attempts among youths. JAMA Psychiatry, 72(11), 1143–1149.

Godoy Garraza, L., Kuiper, N., Goldston, D., McKeon, R., & Walrath, C. (2019). Long‐term impact of the Garrett Lee Smith Youth Suicide Prevention Program on youth suicide mortality, 2006–2015. Journal of Child Psychology and Psychiatry, 60(10), 1142–1147.

Grubinger, T., Zeileis, A. & Pfeiffer, K.-P. (2014). Evtree: evolutionary learning of globally optimal classification and regression trees in R. Journal of Statistical Software, 61(1), 1–29.

Hillis, S. D., Blenkinsop, A., Villaveces, A., Annor, F. B., Liburd, L., Massetti, G. M., et al. (2021). COVID-19–associated orphanhood and caregiver death in the United States. Pediatrics, 148(6), 31–43.

Hoover, S., Lever, N., Sachdev, N., Bravo, N., Schlitt, J., Acosta Price, O., Sheriff, L. & Cashman, J. (2019). Advancing comprehensive school mental health: guidance from the field. Baltimore, MD: National Center for School Mental Health. University of Maryland School of Medicine.

Langley, A. K., Nadeem, E., Kataoka, S. H., Stein, B. D., & Jaycox, L. H. (2010). Evidence-based mental health programs in schools: Barriers and facilitators of successful implementation. School Mental Health, 2(3), 105–113. DOI: 10.1007/s12310-010-9038-1

Leeb, R. T., Bitsko, R. H., Radhakrishnan, L., Martinez, P., Njai, R., & Holland, K. M. (2020). Mental health–related emergency department visits among children aged < 18 years during the COVID-19 pandemic—United States, January 1–October 17, 2020. Morbidity and Mortality Weekly Report, 69, 1675–1680.

Luke, D. A, Calhoun, A., Robichaux, C. B, Elliott, M. B., & Moreland-Russell, S. (2014). The Program Sustainability Assessment Tool: a new instrument for public health programs. Preventing Chronic Disease, 11, 130184. DOI: 10.5888/pcd11.130184. PMID: 24456645; PMCID: PMC3900326

Mattison, E., & Aber, M. (2007). Closing the achievement gap: the association of racial climate with achievement and behavioral outcomes. American Journal of Community Psychology, 40, 1-12. DOI: 10.1007/s10464-007-9128-x.

National Center for School Mental Health (2023). School Health Assessment and Performance Evaluation System. University of Maryland School of Medicine. Available at: https://www.theshapesystem.com/. Accessed May 15, 2023.

National Center on Safe Supportive Learning Environments. (2021). Worksheet: implementing school mental health supports: best practices in action. https://safesupportivelearning.ed.gov/sites/default/files/13-ImpSchMnHlthSprtBtPrt-508_0.pdf

Office of the Surgeon General (2021). Protecting youth mental health: The U.S. Surgeon General’s advisory. U.S. Department of Health and Human Services. Office of the Surgeon General.

Patton, M. (1990). Qualitative evaluation and research methods (pp. 169-186). Beverly Hills, CA: Sage.

SAMHSA. (2023). Understanding child trauma. https://www.samhsa.gov/child-trauma/understanding-child-trauma.

SAMHSA. (2021). Key substance use and mental health indicators in the United States: results from the 2020 National Survey on Drug Use and Health (HHS Publication No. PEP21-07-01-003, NSDUH Series H-56). Rockville, MD: Center for Behavioral Health Statistics and Quality, Substance Abuse and Mental Health Services Administration. https://www.samhsa.gov/data/

U.S. White House Briefing Room. (2022). Fact sheet: Biden-⁠Harris administration announces two new actions to address youth mental health crisis. White House.

Wachino, V., Frank, R. G., Humphreys, K., & O’Brien, J. (2021). The kids are not all right: The urgent need to expand effective behavioral health services for children and youth. USC-Brookings Schaeffer on Health Policy.

Walker, J. S., and Bruns, E. J. (n.d.) Impact of Training and Technical Assistance (IOTTA) For Wraparound. https://nwi.pdx.edu/pdf/IOTTA-results.pdf

Walrath, C., Garraza, L. G., Reid, H., Goldston, D. B. & McKeon, R. (2015). Impact of the Garrett Lee Smith youth suicide prevention program on suicide mortality. American Journal of Public Health, 105(5), 986-993.

Wang, Y., Holt, J. B., Zhang, X., Lu, H., Shah, S. N., Dooley, D. P., et al. (2017). Comparison of methods for estimating prevalence of chronic diseases and health behaviors for small geographic areas: Boston validation study, 2013. Preventing Chronic Disease, 14, E99. https://doi.org/10.5888/

Weinstein, C., Curran, M., & Tomlinson, S. (2003). Culturally responsive classroom management: Awareness into action. Theory Into Practice, 42, 269-276.

Williamson, D. F., Thompson, T. J., Anda, R. F., Dietz, W. H., & Felitti, V. (2002). Body weight and obesity in adults and self-reported abuse in childhood. International Journal of Obesity, 26(8), 1075–1082. 10.1038/sj.ijo.0802038

Yard, E., Radhakrishnan, L., Ballesteros, M. F., Sheppard, M., Gates, A., Stein, Z., et al. (2021). Emergency department visits for suspected suicide attempts among persons aged 12–25 years before and during the COVID-19 pandemic—United States, January 2019–May 2021. Morbidity and Mortality Weekly Report, 70, 888–894.

Yin, R. K. (2003). Case study research: design and methods. 3rd ed. Thousand Oaks, Calif., Sage Publications.

Zhang, X., Holt, J. B., Yun, S., Lu, H., Greenlund, K. J., & Croft, J., B. (2015). Validation of multilevel regression and poststratification methodology for small area estimation of health indicators from the Behavioral Risk Factor Surveillance System. American Journal of Epidemiology, 182, 127-137. https://doi.org/10.1093/aje/kwv002

Zhang, X., Pérez-Stable, E. J., Bourne, P. E., Peprah, E., Duru, O. K., Breen, N., ... & Denny, J. (2017). Big data science: opportunities and challenges to address minority health and health disparities in the 21st century. Ethnicity & Disease, 27(2), 95.

Attachments

1. Implementation Survey (IS)

2. Implementation Key Informant Interview Guide (IKII)

3. Youth and Family Focus Group Moderator Guide (YFFG)

1. Youth (-Y)

2. Family (-F)

4. Collaboration and Partnership Survey (CPS)

5. Training Summary Form (TSF)

6. Participant Feedback Form (PFF)

7. AWARE Pre-Post Training Survey (APPTS)

8. TISS Pre-Post Training Survey (TPPTS)

9. Workforce Follow-up Survey (WFS)

10. Parent Climate and Safety Survey (PCSS)

11. Student Climate and Safety Survey (STCSS)

12. School Staff Climate and Safety Survey (SSCSS)

13. Student Identification and Referral Form (SIRF)

14. AWARE-TISS Online Data Collection System Data Use Agreement (ATODS DUA)

15. Project AWARE-TISS Cross-Site Evaluation Questions

16. Detailed Description of Project AWARE-TISS Cross-Site Evaluation Components

1 For example, an administrative person may register the trainings in the system in advance and the APPTS or TPPTS link can be emailed to the trainer.

2 For example, an administrative person may register the trainings in the system in advance and the PFF link can be emailed to the trainer.

3 For example, an administrative person may register the respondents who provided consent to be contacted in the system in advance and the WFS link can be emailed to the trainer to share with the respondent or emailed to the respondent directly.

4 For example, an administrative person may register the respondents who agreed to participate in the system in advance and the PCSS link can be emailed to the respondent directly or a grantee program staff to share with the respondent.

11