NOW IS THE TIME (NITT)—PROJECT AWARE (ADVANCING WELLNESS AND RESILIENCE IN EDUCATION)-STATE EDUCATION AGENCIES (SEA) NATIONAL EVALUATION
SUPPORTING STATEMENT
The Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Mental Health Services (CMHS) is requesting approval from the Office of Management and Budget (OMB) for new data collection activities for SAMHSA’s evaluation of the Project AWARE (Advancing Wellness and Resiliency in Education)-State Education Agencies (SEA) program, part of the Now Is The Time (NITT) initiative. SAMHSA’s NITT initiative aims to better protect U.S. children from violence by making schools safer and increasing access to mental health services. Programs included are NITT Project AWARE, Healthy Transitions, and two Minority Fellowship Programs (Youth and Addiction Counselors). While SAMHSA’s NITT initiative includes separate programs, these programs are united by their focus on capacity building, system change, and workforce development. The NITT national evaluation is scheduled to be conducted through March 1, 2020, and will conduct a national cross-program evaluation of all three grant programs.
The current request asks the OMB for approval instruments and data collection procedures to support the national evaluation of Project AWARE-SEA grantees. A previous request to conduct site notification and recruitment of Project AWARE-SEA grantees and their school and district partners for the purpose of enlisting sites for participation in the Project AWARE-SEA component of the NITT evaluation was submitted to OMB on December 28, 2015.
The data collection activities described in this package pertain exclusively to the NITT-Project AWARE-SEA program component of the national evaluation. In the fall of 2014, SAMHSA awarded 20 NITT-Project AWARE-SEA grants to 20 geographically diverse states. Each SEA proposed partnerships between at least three high-need Local Educational Agencies (LEAs) to develop a coordinated and integrated plan of services and strategies to address the Project NITT-Project AWARE-SEA goals and objectives. According to the NITT-Project AWARE-SEA solicitation, grantees should increase their in three areas: (1) increase mental health awareness among school-aged (K–12) youth; (2) train those who work with school-aged children to identify and respond to mental health issues in children and youth; and (3) connect children, youth, and families with mental health services. The grants are intended to encourage cross-system collaboration and use of evidence-based strategies to address mental health needs.
NITT-Project AWARE-SEA data collection strategies include telephone interviews, site visits to conduct guided interviews and focus groups, data abstraction from existing surveys and information systems, and Web-based surveys. SAMHSA is requesting approval for data collection for the following nine instruments:
AWARE-SEA Planning and Implementation Activities Inventory (Activities Inventory), to capture information about all activities supported by Project AWARE-SEA resources during the grant period. (Attachment 1)
Project Director Questionnaire, to guide review and input of additional information as needed for state-level activities captured in the Activities Inventory and conducted under Project AWARE-SEA.
Program Coordinator Questionnaire, to guide review and input of additional information as needed for community (or LEA)-level activities captured in the Activities Inventory and conducted under Project AWARE-SEA.
School Coordinator Questionnaire, to guide review and input of additional information as needed for school-level activities captured in the Activities Inventory and conducted under Project AWARE-SEA.
SEA Collaborative Partner Survey (SEA-CPS), to collect information about collaborative processes and partnerships at the state level and to examine the networks involved in successful information sharing and collaborations across child-serving agencies and the families and youth they serve. (Attachment 2)
Local Educational Agency Collaborative Partner Survey (LEA-CPS), to collect information about collaborative processes and partnerships at the local level and to examine the networks involved in successful information sharing and collaborations across child-serving agencies and the families and youth they serve. (Attachment 3)
Collaborative Partner Interview Guide, to collect qualitative information about collaborative processes and partner roles. (Attachment 4)
School Information Systems Data Abstraction Protocol, to compile information from existing school information systems about student socio-demographics, school climate, and school safety. (Attachment 5)
Teacher Mental Health Literacy Survey, to assess the mental health literacy and associated knowledge and skills of teachers in selected schools participating in Project AWARE-SEA activities. (Attachment 6)
Teacher School Climate and School Safety Survey Data Abstraction Protocol, to compile information from existing teacher surveys that include measures of school climate and safety. (Attachment 7)
Student Survey Data Abstraction Protocol, to compile information from existing student surveys that include measures of school climate, school safety, and student coping and resiliency. (Attachment 8)
Student Focus Groups Protocol, to collect qualitative information about student perceptions of school climate; ability to identify signs of mental, behavioral, or emotional health issues; and student knowledge of school- and community-level service access. (Attachment 9)
SAMHSA’s NITT-Project AWARE program is authorized under Section 520A of the Public Health Service Act, as amended, and addresses the Healthy People 2020 Mental Health and Mental Disorders Topic Area (HP 2020-MHMD) and Substance Abuse Topic Area (HP 2020-SA). NITT-Project AWARE also supports three of SAMHSA’s strategic initiatives: (1) increase awareness and understanding of mental and substance use disorders, (2) promote emotional health and wellness, and (3) increase access to effective treatment. Finally, this effort supports the third and fourth components of the President’s NITT Plan, to make schools safer and to increase access to mental health services.
Disruptive and aggressive behavior problems in youth are associated with a host of developmental problems for the perpetrator and victims of such violence (Arseneault et al., 2006). Students’ direct experiences of violence and their perceptions of violence in school are strongly associated with important developmental outcomes, including academic achievement and socio-emotional and behavioral outcomes (Flannery, Wester, & Singer, 2004; Milam, Furr-Holden, & Leaf, 2010). Studies also show strong links between substance use and decreased ability to learn, deficits in academic performance, increases in truancy and dropping out, involvement in delinquency and perpetration of aggression, depressive and anxiety disorders, and suicide ideation or attempts. Effective management of disruptive behaviors in school-aged children is critical for reducing violence in schools and improving child well-being. This involves implementing comprehensive violence and substance abuse prevention programs focused on reducing disruptive and substance abusing behaviors in students and ensuring appropriate facility design and security measures conducive to fostering a safe school environment.
Students who experience personal, adjustment, or mental health problems are at increased risk of academic difficulties, including absenteeism, and reduced academic achievement. Connections with local mental health service providers who coordinate services within school and communities to provide referrals and needed supports to students and families are an essential element for fostering a safe school environment. Unfortunately, only one-fourth of children who need mental health care get the help they need (RAND, 2001). Children and families of minority race/ethnicity may have even less access to mental health services. Unmet mental health needs are particularly high for Latino compared to white children and for uninsured children compared to privately insured children (Kataoka, Zhang, & Wells, 2002). When mental health treatment is received, minorities often receive a poorer quality of care (DHHS, 2001). By virtue of their front-line access to diverse populations of children and families, schools can play an important role in reducing these disparities in access to children’s mental health services. One study found that 70%–80% of children who receive mental health services receive that care in a school setting (Burns et al., 1995). Targeted efforts to reduce disparities by direct in-school service provision or facilitated and coordinated referrals to community-based services can even further enhance the critical role of schools in improving children’s mental health.
Project AWARE builds upon successful strategies identified through the Safe Schools/Healthy Students (SS/HS) Initiative, which reflected the shared vision of three federal departments—Health and Human Services, Education, and Justice—to develop comprehensive and coordinated programs, policies, and service delivery systems to improve the well-being of children and adolescents by enhancing school safety and security and by promoting the mental health of students in communities across the United States. Since 1999, the SS/HS Initiative has supported more than 370 grantees.
From its inception, the SS/HS Initiative required school-community partnerships to support and guide local efforts. Additionally, more recent SS/HS funding created State Management Teams (SMTs) that include state- or tribal-level education, behavioral health, criminal/juvenile justice agencies, and other stakeholders, including youth and families. Multisite SS/HS outcome evaluations have shown reduced violence exposure, increased mental health service access, improved school climate, and enhanced services in SS/HS grantee sites (Trudeau, Williams, & Murray, 2010).
The NITT-Project AWARE-SEA grant program aims to increase awareness of mental health issues among school-aged youth; provide training for school personnel and other adults who interact with youth to detect and respond to mental illness in children and young adults; connect children, youth, and families/caregivers who may have behavioral health issues with appropriate services; and improve conditions for learning and behavioral health outcomes for all school-aged youth (grades K–12).
In the fall of 2014, SAMHSA awarded NITT-Project AWARE-SEA grants to 20 geographically diverse states across the country. Each grantee proposed partnerships with at least three high-need LEAs to develop a coordinated and integrated plan of services and strategies to address the Project NITT-Project AWARE-SEA goals and objectives. High-need LEAs were defined by SAMHSA as those in which at least 10,000, or at least 20% of, children served by the LEA are from families with incomes below the poverty line. These three high-need LEAs must also demonstrate readiness and investment in engaging in these required activities and willingness to work collaboratively with the state in the development and implementation of the comprehensive plan. The partnership between the SEA, the high-need LEAs, and other initiative collaborators is designed to improve and increase access to mental health education, resources, and treatment within each district. Funding will aid in developing and implementing comprehensive programming and services to school-aged youth and the adults who interact with them.
Project AWARE-SEA grantees will plan and implement activities under two components to address three main goals. Component 1, which builds on the SS/HS model, aims to support the enhanced coordination and integration of mental, emotional, and behavioral health services. Component 2 aims to implement Mental Health First Aid (MHFA), Youth Mental Health First Aid (YMHFA), or both at the state and local community levels.
The two SEA grantee goals under Component 1 are to (1) create school communities that promote mental health awareness, well-being, early identification, and early intervention of behavioral health issues through the use of cross-system collaboration, and (2) increase access to and availability of school and community-based mental health services for K–12 students and their families within the partner LEAs. Most SEA grant funds (87%) are set aside for Component 1 activities. These activities can include programs and interventions designed to address outcomes such as improved coordination of services and pooled resources across community partners; improved access to services for students with identified mental, emotional, or behavioral health concerns; and improvements in the safety and climate of schools. SEA grantees may implement evidence-based programs such as Positive Behavioral Interventions and Supports (PBIS), which provides tools and strategies to help educators in elementary, middle, and high schools establish proactive, positive, and instructional schoolwide discipline policies, manage student misbehavior and foster student motivation, and create a positive and productive school climate. Other Component 1 activities may include programs designed to identify and provide services to students at risk of suicide, prevent bullying, and otherwise support a safe and healthy school environment.
Component 2 of Project AWARE-SEA receives the remaining 13% of SEA grant funds to implement MHFA, YMHFA, or both at the state and local community levels. The third goal of Project AWARE-SEA, associated with Component 2, is to train school personnel, emergency first responders, and other adults who interact with school-aged youth to detect and respond to mental illness in children, youth, and young adults. The NITT-Project AWARE-SEA solicitation describes how a lack of awareness of mental health issues on the part of adults can contribute to youths’ not getting needed mental health services. Project AWARE-SEA, therefore, emphasizes the provision of training through MHFA, an evidence-based program. First developed in Australia and now being used in 20 countries, MHFA has both an adult version (MHFA) and a youth version (YMHFA). The adult version addresses mental health issues among those 18 years of age and older. The youth version is for adults who work with adolescents aged 12–18. Both versions introduce participants to mental health risk factors and symptoms, emphasize the importance of early intervention, and use role plays and simulations to teach participants how to assess and respond to individuals experiencing mental health issues. The YMHFA training teaches participants about mental health issues adolescents can experience, including anxiety, depression, substance use, psychotic disorders, disruptive behavior disorders, and eating disorders.
NITT-Project AWARE-SEA grantees will work to create sustainable, integrated programming to build upon the current service capacity for the duration of the grant. Mental health services will be provided to school-aged youth through coordinated efforts led by the SEAs. SEAs will facilitate partnerships between youth-serving organizations to ensure that children, youth, and families may easily access child-centered, community-based resources. Grantees will enhance and improve population data collection capacity and work to better use the data. SEAs will additionally lead efforts to build relationships between other community service systems, such as mental health professions and law enforcement, to ensure a comprehensive approach.
The Project AWARE-SEA evaluation will examine the process, and outcomes of activities by SEA grantees and their LEA and school partners. It will evaluate the capacity of SEAs to effectively involve family and youth, provide a culturally and linguistically competent and family-centered mental health service array, and implement a process for identifying need and delivering services that is informed by data and coordinated across child-serving agencies. Evaluation questions have been developed to understand grantee context, planning, implementation, outputs, and outcomes across each of five NITT priority areas.
The process evaluation will examine the role of initiative partners at the SEA and LEA levels, as well as the activities that grantees undertake to support Components 1 and 2 of the initiative, including both planning and implementation activities. The process evaluation will include qualitative and quantitative components supported by multiple data collection approaches—grantee document review, collaborative partner surveys, site visits, student focus groups, and interviews with initiative partners.
The outcome evaluation will include measures to assess outcomes at the SEA level, the community level for jurisdictions served by the partner LEAs, and the school level for those schools participating in Project AWARE-SEA within the partner LEAs. Data sources to inform the outcome evaluation were designed to capitalize on school-level data collection already in place and to minimize burden on AWARE-SEA grant partner systems and schools. Outcome evaluation data sources include data abstraction protocols to compile and analyze existing data from school information systems, teacher school climate and school safety surveys, and student surveys. The outcome evaluation will also be informed by primary data collection through teacher mental health literacy surveys and student focus groups, as well as data collected from other evaluations supported by SAMHSA. These data sources are designed to assess change in SEA-, LEA-, and school-level outcomes, including improvements in
mental health literacy among students, teachers, and other adults who interact with youth;
early and accurate identification of mental health and behavioral needs;
access to child-centered, family-focused, and culturally and linguistically competent mental health services;
school safety;
school climate; and
student coping and resiliency skills.
The outcome evaluation will assess changes within all SEAs and LEAs over the course of the initiative. It will also include an additional comparative component for a subset of AWARE-SEA grantees, in which schools participating in Project AWARE-SEA activities will be matched with similar schools to further examine the impact of grantee activities on key outcomes. This comparison group substudy will include selected grantees with student-level surveys already in place that measure key student coping and resiliency, school climate, and school safety measures. In these selected LEAs, comparison schools will be identified that are either not prioritized for Project AWARE-SEA activities during the grant period, or who will be implementing AWARE-SEA activities later in the grant period. Existing data collection instruments will allow analysis of AWARE-SEA outcomes between treatment and comparison schools with minimal burden on schools and LEAs.
The NITT-Project AWARE-SEA national evaluation will capture key elements of collaboration, coordination, and integration at the SEA and LEA levels that increase adults’ ability to identify and appropriately respond to mental, behavioral, and emotional problems in school-aged youth; provide for youth access and engagement in services; and promote safety and wellness among schools and students. Ultimately, answers to the NITT-Project AWARE-SEA specific evaluation questions (see Table 1) will feed into the integrated evaluation model to address the success of the full SAMHSA NITT initiative in achieving its prime mission of making schools safer and increasing access to mental health care.
The national evaluation will also include required grantee performance measures submitted by NITT-Project AWARE SEAs. Project AWARE-SEA grantees are required to collect data on six performance measures. Data for three of these measures are collected and reported quarterly at the SEA level. These measures include:
The number of individuals who were trained as (A) MHFA and YMHFA First Aiders and (B) MHFA or YMHFA Instructors during the previous 3 months that are not part of the mental health workforce;
The number of adults in the mental health workforce at both the SEA- and LEA-levels who participated in (A) the MHFA/YMHFA Instructor training and (B) the MHFA/YMHFA First Aider training during the previous 3 months; and
The number of individuals referred to mental health or related services.
Data for the remaining three measures are reported annually by the participating LEAs to the SEA who, in turn, reports this information to the SAMHSA Project Officer in the annual and final performance reports. These annual performance measures include:
The number of school-aged youth served as a result of implementing strategies identified in the SEA comprehensive plan;
The total number of school-aged youth who received school-based mental health services; and
The percentage of mental health service referrals for school-aged youth which resulted in mental health services being provided in the community.
These SEA- and LEA-level measures will be made available to the national evaluation team and will not need to be collected through the surveys and interviews at the center of this OMB application.
The core NITT AWARE-SEA evaluation questions to assess program processes and outcomes fall into five priority areas: 1) collaboration and coordination, 2) mental health service system capacity and infrastructure, 3) early and accurate identification of mental health needs, (4) mental health service access for individuals in need, and 5) individual resilience and functioning and school/community safety. Table 1 describes the NITT AWARE-SEA evaluation questions organized by these priority areas. These evaluation questions will be addressed in the data collection procedures section as described in Section A.2.
Table 1. Evaluation Questions for the Now Is the Time (NITT)-Project AWARE-SEA Program
|
|
EQ-A1. |
To what extent do NITT activities lead to enhanced coordination and collaboration across child- and youth-serving agencies? |
EQ-A2. |
What are the barriers to and facilitators of state- or local-level collaboration, partnership development, and shared decision making? How were they addressed? |
EQ-A3. |
To what extent are families and youth involved, and what impact does their involvement have? |
EQ-A4. |
What strategies effectively engage families in connecting with schools and communities? |
EQ-A5. |
What is the effect of collaboration across federal, state, and local agencies on NITT-Project AWARE-SEA grant activities, processes and outcomes? |
|
|
EQ-B1. |
How are state- and local-level systems changing in response to NITT-Project AWARE-SEA activities? |
EQ-B2. |
To what extent is information sharing across child-serving agencies enhanced as a result of NITT? |
EQ-B3. |
What services are being (or will be) provided as a result of the NITT Initiatives (including modality, type, intensity, and duration)? |
|
|
EQ-C1. |
What, if any, increases do youth see in the knowledge, skills, and abilities of school officials and the variety of actors in the public sphere that interact with youth to detect and respond to signs of mental illness in children and adolescents as a result of NITT-Project AWARE-SEA? |
(continued)
Table 1. Evaluation Questions for the Now Is the Time (NITT)-Project AWARE-SEA Program (continued)
EQ-C2. |
What is the effect of NITT-Project AWARE-SEA at the state and community levels on mental health literacy? |
EQ-C3. |
To what extent are mental health (and co-occurring substance use) problems identified among children and youth earlier than before NITT-Project AWARE-SEA? |
|
|
EQ-D1. |
What is the effect of NITT-Project AWARE-SEA on referrals to mental health services? Are children and youth referred and linked to mental health services that are community based, family focused, and culturally and linguistically competent? |
EQ-D2. |
To what extent do the NITT activities collectively increase access to mental health services for children and youth? |
|
|
EQ-E1. |
To what extent does NITT-Project AWARE-SEA lead to improved coping and resiliency among students? |
EQ-E2. |
What is the effect of NITT-Project AWARE-SEA on school climate? |
EQ-E3. |
What are the effects of NITT-Project AWARE-SEA on school safety, including outcomes such as underage drinking, school violence, bullying, and perceptions of safety (for students and teachers)? |
EQ-E4. |
To what extent do the NITT activities collectively contribute to making schools and communities safer? |
EQ-E5. |
Do schools prioritized for AWARE-SEA implementation activities demonstrate improvements in school safety, school climate, and student coping and resiliency compared to matched schools in the same LEA? |
The NITT-Project AWARE-SEA process and outcome evaluation is designed to answer the evaluation questions outlined in Table 1 and described in more detail in the sections that follow. The process evaluation will examine the role of initiative partners at the SEA and LEA levels, as well as the activities that grantees undertake to achieve AWARE-SEA objectives, including both planning and implementation activities. The process evaluation will include qualitative and quantitative components supported by multiple types of data collection—grantee document review, collaborative partner surveys, and collaborative partner interviews. The outcome evaluation will include measures to assess outcomes at the SEA level, the community level for jurisdictions served by the partner LEAs, and school-level outcomes for those schools participating in Project AWARE-SEA within the partner LEAs. Data sources to inform the outcome evaluation were designed to capitalize on school-level data collection already in place and to minimize burden on LEA AWARE-SEA grant partners. Outcome evaluation data sources include data abstraction from existing student- and teacher-level surveys, performance measures submitted to SAMHSA, data abstraction from school information systems, a teacher mental health literacy survey, student focus groups, and program participant survey data collected through another SAMHSA-funded evaluation. These data sources are designed to assess change in SEA-, LEA-, and school-level outcomes, including improvements in
mental health literacy among students, teachers, and other adults who interact with youth;
early and accurate identification of mental health and behavioral needs;
access to child-centered, family-focused, and culturally and linguistically competent mental health services;
school safety;
school climate; and
student coping and resiliency skills.
The NITT-Project AWARE-SEA national evaluation will use a pre/post design, augmented by a matched comparison school design in a selected group of AWARE-SEA grantees (discussed further in Section A.16, Analysis). It will combine qualitative and quantitative data and methodologies to fully address SAMHSA’s objectives for the national evaluation. Feedback on the NITT-Project AWARE-SEA evaluation design and data collection instruments (see Section A.8 and Section B.5) has been solicited from internal SAMHSA staff, grantee-level project directors and project evaluators, and an expert advisory panel. After careful review, revisions were made to streamline the instruments, reduce burden, simplify wording, increase variation in response options, improve coherence of scales for summing, and create consistency in assessing key outcomes among teachers, other adults who interact with children and youth, and students.
The following section describes each NITT-Project AWARE-SEA national evaluation instruments for which SAMHSA is requesting approval. Some instruments that will inform the national evaluation, including the MHFA participant survey and the performance measures and other information submitted by grantees to SAMHSA as part of the grant requirements, and not included here as they are covered under separate OMB clearances.
A key component of the process and outcome evaluations is to document what the SEA grantees plan and implement under Project AWARE-SEA, and how those activities were associated with observed outcomes. The AWARE-SEA Activities Inventory and supporting questionnaires will capture all activities and associated outputs supported by Project AWARE-SEA resources. The Activities Inventory will be compiled by the national evaluation team from existing grantee reports and documentation, and then reviewed for accuracy and completeness annually through questionnaires administered to the SEA project director and/or SEA coordinator, each LEA coordinator, and each participating school program coordinator.
At the SEA level, the Activities Inventory will focus on activities that are primarily related to planning, coordination, and collaboration among SEA-level partners, as well as the three SEA-level performance measures. At the LEA level, the Activities Inventory will focus on activities conducted in the jurisdiction served by each LEA, in schools and with local partner organizations, such as mental health service providers and juvenile/criminal justice agencies. These activities are primarily related to coordination between local partners and implementation of programs and activities designed to improve awareness of mental health needs, service access, service system capacity, and overall school climate and safety. At the community and school levels, the Activities Inventory will document activities designed to achieve AWARE-SEA objectives in individual programs (e.g., mental health service provision to school-aged youth) and schools.
The evaluation team will abstract and update as much of this information as possible from existing grantee documentation submitted to SAMHSA. The Activities Inventory will be available to core project staff (i.e., the questionnaire respondents described above) through the AWARE-SEA Web-based data collection portal. This information will then be confirmed and/or updated as indicated through questionnaires conducted with the SEA project director, the LEA project coordinators, and points of contact at each participating school. This questionnaire is intended to confirm the accuracy and completeness of the information in the Activities Inventory, and to update it as needed. A series of questions will be asked about each activity, including the name and description of the activity; partners involved in the implementation; the target population; the frequency, mode, and duration of the activity; the number and characteristics of the participants; and the obstacles or facilitators to implementation. The interview will also aim to measure the engagement of parents and families, in addition to any partners involved in the activity.
The AWARE-SEA Activities Inventory will be reviewed annually by core project staff and updated through the questionnaires. These questionnaires will be administered on an annual basis as soon as OMB approval is received (currently estimated for July 2016).
The Project AWARE model draws from successful strategies identified through SAMHSA’s Safe Schools/Healthy Students Program, which rely on collaboration at the state and local levels to plan and implement activities in support of program goals of enhanced mental health literacy among adults who interact with youth, increased access to services, and improved school safety, school climate, and student resiliency. This collaboration is also critical to achieve interim outcomes, such as improved service integration and information sharing among the systems that serve school-aged youth. SEA-CPS is designed to capture collaborative partnerships to examine the networks involved in successful information sharing and collaborations across child-serving agencies and the extent of family and student involvement. SEA-CPS will be administered to all identified collaborative partners at the state level, including the SEA project director and project coordinator; project coordinators from each of the partner LEAs; youth and family representatives; and partners representing state mental/behavioral health, state criminal/juvenile justice agencies, and other identified partner agencies at the state level, such as state Medicaid directors and child welfare, early childhood, and faith-based organizations. SEA-CPS will consist of two modules drawn from widely used surveys that tap into the constructs of interest for the national evaluation. These constructs include:
collaborative efforts to plan and implement AWARE-SEA activities;
coordination, integration, and information sharing across AWARE-SEA partners;
obstacles and facilitators to collaboration;
role of partner systems and individuals participating in collaborative efforts;
collaborative organization;
collaborative processes;
collaborative level of partner investment; and
resources to support AWARE-SEA goals.
The SEA-CPS will be administered annually as soon as OMB approval is received (currently estimated for July 2016).
The LEA-collaboration and coordination survey will be administered to approximately 20 local initiative partners in each participating LEA, including the project coordinator and representatives from each key partner agency, family or youth perspective representative, program provider, and school. The national evaluation team will collect LEA-collaboration and coordination survey data through the NITT Web-based data collection and repository system. The LEA-collaboration and coordination survey will include an LEA analogous instrument similar to Module 2 from the SEA-CPS, which focuses on interprofessional team collaboration. The overall LEA-collaboration instrument will not include Module 1 from the SEA-CPS and only include this 59 item survey to minimize burden on local partners and to focus specifically on information sharing and coordination among youth-serving agencies in the community. The LEA-collaboration and coordination survey includes measures of mental health service system capacity and infrastructure. The survey will be administered twice during the grant period, as soon as OMB approval is received (expected in July 2016) and in the last year of the SEA-AWARE grant period (estimated for July 2019).
The SEA-CPS will be supplemented by collaborative partner interviews with a sample of the CPS respondents. The interviews will include more open-ended questions about barriers to and facilitators of state- and local-level collaboration, partnership development, and shared decision making; how the barriers were addressed; and how the collaboration and coordination enhanced or influenced each of the individual NITT AWARE-SEA goals and objectives. These interviews will also explore the planning and implementation activities under way, the barriers and facilitators encountered, and the perceived effects of those activities on agency policy and practice. The data collected during these interviews will also be used to inform the Activities Inventory described above. Collaborative partner interviews will be conducted annually beginning as soon as OMB approval is received (estimated to be July 2016) and continuing until July 2019.
All AWARE-SEA grantee communities have some tool in place to either collate or maintain student-related data. The school information systems data abstraction protocol will provide a guide for SEAs and their LEA partners to upload information from the school information systems to the national evaluation web-based data collection platform. These school-level measures will be used to answer outcome evaluation questions focused on school safety and school climate. Specific constructs that the national evaluation team is interested in abstracting from these information systems include student administrative data, such as the attendance and graduation rates of the school; demographic data, including the race/ethnicity and grade-level counts; and indicators of school climate and safety, such as counts of drug-, weapons-, bullying-, and fighting-related incidents, and counts of suspension and expulsion incidents.
The data abstraction protocol will detail the procedure through which the national evaluation team will abstract data from each LEA or school information system. The data or summary information from the school information systems will be transferred through the AWARE-SEA secure Web-based data collection and transfer portal. These data will be requested annually beginning as soon as OMB approval is received and then collected both retrospectively and prospectively to include data from school years 2014-2015, 2015-2016, 2016-2017, 2017-2018 and 2018-2019.
The national evaluation team developed a survey to assess the mental health literacy and associated knowledge and skills of teachers in a sample of schools participating in AWARE-SEA. These instruments are critical to assessing the extent to which the teachers who interact with youth on a daily basis are able to recognize and effectively respond to signs of mental, behavioral, or emotional health needs. The survey will also measure teacher knowledge of and participation in school-level programs designed to enhance mental health literacy or promote a safe and supportive school environment. This survey will be administered twice to teachers in a random sample of schools in partner LEAs (once in the fall of the 2016-17 school year and again in the spring of the 2018-2019 school year), stratified by school type and size.
The national evaluation team developed a data abstraction protocol and secure data transfer platform to collect information from the existing student and teacher surveys. These survey data will be provided by the SEA and LEAs from instruments already in place in a sample of schools in all LEAs and from identified comparison schools in comparison group substudy LEAs. The national evaluation will rely on existing student survey measures of school climate, school safety, mental health, and coping/resiliency. Aggregate data will be abstracted from LEAs that currently have instruments in place that include key outcomes of interest and have indicated that these data can be made available to the AWARE-SEA national evaluation. The Center for Disease Control’s Youth Risk Behavior Surveillance System (YRBSS) will be a primary data source, in addition to surveys developed and in use by individual states. The data abstraction protocol will be customized for each SEA based on the specific data collected by each state. Key AWARE-SEA outcomes measures collected through existing student surveys include:
School climate: staff-student relationships, student-student relationships, student connectedness to school, problems posed by student risk behavior, student academic achievement, clarity of expectations, respect for diversity, fairness of rules, student engagement, student autonomy;
School safety: perceptions of school safety, victimization, physical fights, other delinquent behaviors (e.g., tobacco use, alcohol use, other illegal drug use);
Mental health: sadness/hopelessness, signs of suicide, and anxiety; and
Coping/Resiliency: self-regulation/self-control and self-confidence.
These data will be requested annually beginning as soon as OMB approval is received and then collected both retrospectively and prospectively to include data from school years 2014-2015, 2015-2016, 2016-2017, 2017-2018 and 2018-2019.
Not all LEAs have existing student surveys that include measures of coping and resiliency, school climate, and school safety. Furthermore, most do not collect information about student knowledge of resources or programs to support a safe and healthy school environment, or knowledge of available services to respond to identified mental health needs. To address these information gaps, the national evaluation team will conduct student focus groups in one of the three partner LEAs for each SEA grantee. Within the identified LEA, the focus groups will be conducted in four prioritized AWARE-SEA schools for all LEAs. The evaluation team will identify the LEA in collaboration with the SEA grant leadership, ideally defined as the LEA with the most AWARE-SEA implementation activities documented in the Activities Inventory (and presumably, the LEA with the greatest need at the beginning of the grant). The evaluation team will work with the project evaluation team, LEA coordinator and school coordinator to gain local Institutional Review Board (IRB) approval and informed consent to conduct these focus groups during site visits, conducted once in the 2016-17 school year and again during the 2018-2019 school year). The guided discussion protocol will focus on participants’ general knowledge of available resources, programs to support AWARE-SEA activities, and overall perceptions of school climate and safety.
Focus groups will be conducted with students to supplement the information obtained from the existing student survey measures. Focus groups will also collect information on students’ knowledge of programs in place to achieve AWARE goals. A copy of the semistructured focus group protocol is included in Attachment 9. Key topics include:
knowledge of programs or resources at the school to promote mental health literacy, a positive school climate, or a safe school environment;
ability to identify and respond to symptoms of mental, behavioral or emotional needs;
knowledge of available programs to meet student mental health needs;
overall school climate; and
overall school safety
The multiple data collection methods and sources included within the NITT-Project AWARE-SEA national evaluation are necessary to address the core NITT-Project AWARE-SEA evaluation questions. These questions require that information be gathered from individuals representing various layers of the NITT-Project AWARE-SEA grantee community (SEA, LEA, community-based service providers, schools, teachers and students). The evaluation questions also require that multiple types of individuals report upon key outcomes of interest (project directors [SEA-level], collaborative partners, including family and youth partners, program coordinators [LEA-level], school coordinators, teachers and other adults who come into contact with students, and students). Table 2 provides an overview of the NITT-Project AWARE-SEA evaluation priority areas, main constructs of interest and the data sources.
Table 2. NITT-Project AWARE-SEA Data Collection Procedures by Evaluation Component and Key Measures
Data collection level/ instrument |
Evaluation Priority Area |
Key Measures |
||||
A |
B |
C |
D |
E |
||
SEA-level instruments |
||||||
Coordination and Integration Plan review* |
|
|
|
|
|
|
Progress report review* |
|
|
|
|
|
|
Project evaluation findings review* |
|
|
|
|
|
To be determined from each grantee’s evaluation plan |
SEA AWARE activities inventory |
|
|
|
|
|
|
SEA collaboration and coordination survey |
|
|
|
|
|
|
Collaborative partner interviews |
|
|
|
|
|
|
LEA-level instruments |
||||||
LEA AWARE activities inventory |
|
|
|
|
|
See SEA AWARE activities inventory |
LEA collaboration and coordination survey |
|
|
|
|
|
See SMT collaboration and coordination survey |
(continued)
Table 2. NITT-Project AWARE-SEA Data Collection Procedures by Evaluation Component and Key Measures (continued)
Data collection level/ instrument |
Evaluation Priority Area |
Key Measures |
||||
A |
B |
C |
D |
E |
||
Program-level instruments |
||||||
Program Coordinator Questionnaire |
|
|
|
|
|
See SEA AWARE activities inventory |
Program participant survey* [NOTE: This survey is being conducted by another SAMHSA-funded evaluation effort] |
|
|
|
|
|
|
School-level instruments |
||||||
School Coordinator Questionnaire |
|
|
|
|
|
|
School Information Systems Data Abstraction |
|
|
|
|
|
|
Teacher Mental Health Literacy Survey |
|
|
|
|
|
|
(continued)
Table 2. NITT-Project AWARE-SEA Data Collection Procedures by Evaluation Component and Key Measures (continued)
Data collection level/ instrument |
Evaluation Priority Area |
Key Measures |
||||||||
A |
B |
C |
D |
E |
||||||
Teacher School Climate and School Safety Survey Data Abstraction |
|
|
|
|
|
|
||||
Student Survey Data Abstraction |
|
|
|
|
|
|
||||
Student Focus Groups |
|
|
|
|
|
|
||||
* The burden associated with these instruments is covered under a separate OMB approval. Evaluation Priority Areas |
||||||||||
A. Collaboration and coordination |
B. Mental health service system capacity and infrastructure |
C. Early & accurate identification of mental health needs |
D. Mental health service access for individuals in need |
E. School/ community safety and resiliency |
||||||
The following are self-administered, Web-based surveys to be completed through the NITT-Project AWARE-SEA online data collection system:
SEA-CPS and LEA-CPS;
Project Director, LEA Coordinator, and School/Program Coordinator Web Questionnaires; and
Teacher Mental Health Literacy Survey
Using a Web instrument allows for automated data checks as well as for skip procedures that will reduce the burden among respondents and possibility of data entry error, thereby increasing the efficiency of data entry and improving data quality. The automated data checks will help respondents give valid responses (e.g., restricting the range of responses when asked about “number of days events occurred in the past year” to 0 to 365), and also ensure that responses follow the expected format (e.g., numbers or dates where those are expected). Responses will generate skip patterns for later questions in the instrument, where the respondents only complete relevant sets of questions and do not see others.
Using a Web-based system will provide the capability to send automatic e-mail reminders to grantees if and when surveys have not been completed.
The web-based system will also facilitate secure, data transfer for all data abstraction protocols, which will collect information annually from existing school information systems, student surveys, and teacher surveys that measure school climate and school safety.
A dashboard and other reports will also be available to SAMHSA and the NITT-Project AWARE-SEA team, as well as the grantees, to monitor the overall status of data collection. Grantees will have access to aggregated statistics from their own data and will also be able to review online descriptive information from the Activities Inventory.
The qualitative instruments include collaborative partner interviews and student focus group moderator guides administered as part of the NITT-Project AWARE-SEA data collection. Respondents will be read questions by a NITT-Project AWARE-SEA evaluation team liaison while a NITT-Project AWARE-SEA note taker records each response. With respondent consent, interviews will be recorded as a back-up to the note taker. After the interview, the interviewer and note taker will review the completed interview notes for accuracy; any areas of discrepancy will be validated with the recording. Once the interview responses are considered final, the recording will be deleted. Until they are deleted, the recordings will be kept by the note taker on a secure, password-protected computer.
The Project AWARE-SEA national evaluation team developed a detailed crosswalk to match constructs and measures required to answer the NITT-Project AWARE-SEA and NITT-consolidated evaluation questions with the measures and instruments already in place in the 20 SEA grantees and their partner LEA communities. This crosswalk identified a number of measures where data collection was already in place, and the national evaluation was planned to avoid duplicative data collection efforts and to minimize burden on Project AWARE partners at the SEA, LEA, and school levels. The Project AWARE-SEA national evaluation will therefore incorporate measures from several existing documents and data collection efforts into the evaluation design, including:
quarterly SEA-level performance measures submitted to SAMHSA;
annual GPRA measures submitted to SAMHSA as part of performance reports;
document review and data abstraction from existing plans and reports, including the Coordination and Integration Plan, the Implementation Plan, annual performance reports, and project evaluation findings;
secondary data abstraction of measures compiled from school information systems;
secondary data abstraction of measures collected through existing teacher surveys measuring school climate and school safety; and
secondary data abstraction of measures collected through existing student surveys.
As described above, SAMHSA monitors the performance of NITT-Project AWARE-SEA programs by requiring the grantees to collect and submit SEA-level performance measures quarterly and LEA-level performance measures annually through progress reports. The Project AWARE-SEA national evaluation will incorporate data from the six required performance measures.
The AWARE-SEA Activities Inventory is a key component in answering several Project AWARE-SEA evaluation questions. The Activities Inventory measures will be abstracted by the evaluation team from the grantee coordination and integration plan and annual progress reports, then reviewed and updated as necessary during annual questionnaires conducted with the grant leadership and coordinators.
Nearly all LEAs involved in the Project-AWARE SEA initiative have some student and teacher surveys already in place, which include measures of student coping and resiliency, school climate, and school safety. These data will be collected by the national evaluation team and harmonized to answer evaluation questions under priority area 5. Teacher surveys include measures of teacher perceptions of school safety, school climate, student engagement, and student coping and resiliency. The existing LEA student surveys largely include measures of students’ perceptions of school safety, school climate, and engagement with their peers and with adults in the school. A smaller number of existing surveys include measures of student coping and resiliency. The national evaluation team will also conduct student focus groups in selected LEAs where the existing student survey data do not include many of the key measures to answer outcome evaluation questions. The focus groups will be used to collect information about student knowledge of programs to achieve AWARE goals, participation in those programs, mental health literacy knowledge and skills, and perceptions of school climate and school safety.
In addition to incorporating data from these existing sources to minimize duplication and burden on grantee communities, the Project AWARE-SEA national evaluation will enter into a data-sharing agreement with another SAMHSA-funded evaluation to incorporate additional measures of mental health literacy and associated knowledge, skills and abilities into the design. A separate contract supports the national evaluation of the Mental Health First Aid program and will collect measures of the knowledge, skills, and abilities of MHFA program participants to detect and appropriately respond to signs of mental illness in students. The Project AWARE-SEA national evaluation will incorporate these data to answer key questions about the effect of AWARE-SEA activities on the abilities of adults in the grantee communities to identify and respond to youth with mental health, emotional, or behavioral problems.
Participation in this evaluation will not impose a significant impact on small entities. NITT-Project AWARE SEA grantees and the LEA, school, and program partners may include state agencies, local jurisdictions, community service providers, and schools. Some of the community service providers may be small entities; however, the NITT-Project AWARE-SEA data collection instruments are designed to include only the most pertinent information needed to be able to carry out the evaluation effectively, and their impact will not be significant.
The multiple data collection points for the Collaborative Partner Surveys; Project Director, Project Coordinator, Program Coordinator, and School Coordinator Questionnaires; Teacher Mental Health Literacy Surveys; and Student Focus Groups are necessary to track and evaluate grantees’ progress and change over time. Without multiple data collection points, SAMHSA would be unable to adequately evaluate the NITT-Project AWARE-SEA initiative and measure progress and change over time associated with these grantees and programs and services for this child and youth population. The NITT-Project AWARE-SEA evaluation has made every effort to ensure that data are collected only when necessary and that extraneous collection will not be conducted.
This information collection fully complies with the guidelines in 5 CFR 1320.5(d)(2).
The notice required by 5 CFR 1320.8(d) was published in the Federal Register on April 21, 2016 (81 FR 23501). No comments were received.
SAMHSA has made extensive use of experts in the area of children and youth mental health research and current grantee staff (e.g., project directors, local evaluators) to provide guidance on the design and analysis of the NITT-Project AWARE-SEA evaluation. The following Project AWARE-SEA in-person and virtual meetings were held to review the various aspects of the NITT-Project AWARE-SEA evaluation, including the preliminary evaluation plan and data collection procedures, which were distributed prior to each meeting to give participants the opportunity to review and formulate feedback:
introductory webinar (June 2015),
grantee meeting (July 2015),
evaluator virtual meeting (August 2015), and
expert panel meeting (August 2015).
The list of expert panel members is provided in Table 3.
Table 3. Now is the Time—Project AWARE-SEA Expert Panel
Expert |
Affiliation |
Contact Information |
Denise Gottfredson, PhD |
Professor Department of Criminology and Criminal Justice, University of Maryland at College Park |
Email: [email protected] |
MaJosé Carrasco, MPA |
Director |
Email: [email protected] |
Lynda Gargan, |
Interim Executive
Director |
Phone: 240-403-1901 Email: [email protected] |
Susan Keys, PhD |
Associate Professor and
Senior Researcher |
Phone: 541-322-2046 Email: [email protected] |
Benjamin Le Cook, PhD, MPH |
Assistant Professor of
Psychiatry |
Phone: 617-503-8449 Email: [email protected]; [email protected] |
Pamela Orpinas, PhD |
Professor |
Phone: 706-542-4372 Email: [email protected] |
Mark Salzer, PhD |
Professor and
Chair |
Phone: 215-204-7879 Email: [email protected] |
Grantees and project evaluators have had several opportunities to provide feedback on the preliminary evaluation plan and data collection procedures. Feedback was provided on all aspects of the evaluation and participants’ comments and suggestions were incorporated into the development of the data collection procedures and instruments. Feedback included:
Create opportunities to provide NITT-Project AWARE-SEA grantees with feedback from the program evaluation throughout the evaluation period.
Perform a crosswalk of the Project AWARE-SEA–required performance measures and existing data collection approaches in grantee communities to ensure a lack of duplication and unnecessary burden on partner community systems and schools.
Offer teachers incentives for their participation in the teacher mental health literacy survey.
Teacher Mental Health Literacy Survey: SAMHSA will provide a $20 gift card to each teacher who completes the Web-based teacher survey at each administration. This amount is consistent with the amounts used in current National Center for Education Statistics (NCES) studies, such as the Early Childhood Longitudinal Study, Kindergarten Class of 2010-2011. While our stress in school contacts will be on the invaluable nature of the research (the fact that so much can be learned in what is a neglected area of research and investigation is indeed a selling point), incentives can many times be an effective tool (especially but not exclusively at the individual level)—assuming OMB’s continued receptivity to the use of monetary incentives in this setting. Teachers in particular are reluctant to take on additional voluntary activities that are not written into their job description.
We will give each teacher an incentive each time that he or she completes the teacher survey. This incentive will be $20 in accordance with the amount stated in the Guidelines for a survey of medium burden. This incentive will be instrumental in obtaining completed teacher surveys for this study because teachers are the target of numerous requests to complete surveys on a wide variety of topics from state and district offices, independent researchers, and the Department of Education (Policy and Program Studies Service and NCES). Further, the teachers’ school days are already quite busy, potentially requiring them to complete surveys outside school time. There are also in some localities collective bargaining agreements that do not allow teachers to complete surveys during school time.
Comparison School Participation: An important component of the NITT-Project AWARE-SEA evaluation is a matched school comparison study which will be conducted in a subset of the LEAs to minimize burden on grantees. The evaluation team will work to identify appropriate comparison schools in the LEAs participating in Project AWARE-SEA and recruit those schools to participate in the comparison group substudy. The substudy will require all treatment and comparison schools to complete annual surveys about programs in place to achieve AWARE goals, coordinate with the national evaluation team to administer a mental health literacy survey with a sample of teachers in the school, and assist in recruiting students to participate in focus groups. The identified point of contact (or multiple points of contact) in each treatment and comparison school will also work with the national evaluation team to provide access to the school information system and existing surveys that measure school climate and school safety. These measures will allow the comparison of Project AWARE (“treatment”) schools with comparison schools regarding change in school-level mental health literacy, mental health service access and receipt, student coping skills and resiliency, school climate, and school safety.
To assist in recruitment and increase participation rates, a $250 gift card will be provided to each comparison school for each year of their participation, $50 of which will go to the coordinator responsible for gathering and reporting data (where allowed by school regulations). The gift card will also serve as a thank you for the time staff in the school invest as these schools are not included in NITT-Project AWARE-SEA grant activities and thus do not benefit from professional development, program implementation support, or technical assistance from the state or district related to programs. These incentive amounts are comparable to those provided in other OMB-approved studies and evaluations in similar educational settings with similar data collection approaches (e.g., respondent type, burden), for example, the Progress in International Reading Literacy Study (OMB No. 1850-0645), the High School Longitudinal Study Field Test First Follow Up and Main Study (OMB No. 1850-0852), and the Middle Grades Longitudinal Study–Item Validation Field Test (OMB No. 1850-0911).
No other incentives or gifts will be given to any respondents.
Concern for privacy and protection of respondents’ rights will play a central part in the implementation of all study components. The NITT-Project AWARE-SEA evaluation team members developing the instruments and surveys and analyzing the data have extensive experience protecting and maintaining the privacy of respondent data. All NITT-Project AWARE-SEA data will be securely stored on a protected server. A SORN/PIA is currently being sought.
Project director, Project coordinator, Key stakeholder, and School coordinator interviews. Prior to beginning any interview, the participants, including both project staff and key informants, will be read, asked to sign, and provided a copy of the consent form. The consent form informs participants of their rights, including the right to not answer any question; participants must provide written consent to before the interview begins. If a participant does not provide written consent to participate then the interview will not take place. Participants will also be asked for their consent to record the discussion. Recordings will be used to ensure that information is correctly captured from multiple interviews, information has been consistently captured across interviewees, and to correct and clarify brief written notes as needed and as part of data quality assurance procedures. Recordings will only be accessible to the contractor and will be stored on password-protected secure servers and destroyed once de-identified notes are completed. No individual-level or personal data will be collected by the NITT-Project AWARE-SEA evaluation from grantee staff, who will only provide information about their organizations, NITT-Project AWARE-SEA activities, and implemented interventions, rather than information about themselves personally.
Student Focus Groups. Prior to beginning the Student Focus Group discussions, conducted during the Grantee Site Visit, written consent forms will be distributed and collected from parents of focus group participants. Participants will be asked to provide verbal assent to participate in the focus groups. The consent and assent materials inform participants of their rights, including the right to not answer any question or participate in any discussion; participants must provide written consent to participate before the focus group discussion begins. If a participant does not provide written consent to participate then that individual will be excused from the group. Participants will also be asked for their consent to record the discussion. Recordings will be used to ensure that information is accurately captured and to correct and clarify written notes as needed as part of data quality assurance procedures. Recordings will only be accessible to the contractor and will be stored on password-protected secure servers and destroyed once de-identified notes are completed.
All information collected during the interviews and focus groups will be reported in aggregate only. Grantee partners will be identified as participating in the interviews, but no individual respondents will be linked to any information provided during those interviews. Student focus groups participants will not be identified at any time or linked to any information discussed during the focus groups.
All other data collection instruments collect programmatic data (i.e., information about the organizations and implemented interventions) at the grantee, community, and school levels along with aggregated, non-identifying school- and participant-level data (e.g., student survey responses aggregated to the grade level, estimated percent of individuals receiving specific types of intervention approaches). Sensitive respondent information, such as birthdates and social security numbers, will not be collected.
To create a profile for grantee, community, and school staff to log in to the NITT-Project AWARE-SEA online data collection system, the system will collect names, telephone numbers, mailing addresses and e-mail addresses of grantee, community, and school staff. This identifying information will be accessible only to select NITT-Project AWARE-SEA evaluation staff. No other personal information will be collected from participants as the focus of this data collection is on programmatic characteristics, including the processes and outcomes of the NITT-Project AWARE-SEA grantees.
The NITT-Project AWARE-SEA systems development team takes responsibility for ensuring that the Web and data systems are properly maintained and monitored. Server staff will follow standard procedures for applying security patches and conducting routine maintenance for system updates. Data will be stored on a password-protected server, and access to data in the system will be handled by a hierarchy of user roles, with each role conferring only the minimum access to system data needed to perform the necessary functions of the role.
For all data collection activities, the contractor will use passwords to safeguard all project directories and analysis files containing completed survey data to ensure that there is no inadvertent disclosure of study data. Contractor staff has been trained on handling sensitive data and the importance of privacy. All contractor staff will sign a privacy pledge. In addition, interviews and focus group guides and all consent forms will be reviewed and approved by the contractor’s IRB (Federal Wide Assurance #3331). In keeping with 45 CFR 46, Protection of Human Subjects, the NITT-Project AWARE-SEA procedures for data collection, consent, and data maintenance are formulated to protect respondents’ rights and the privacy of information collected. Strict procedures will be followed for protecting the privacy of respondents’ information and for obtaining their informed consent.
Data from the NITT-Project AWARE-SEA client interviews will be safeguarded in compliance with the Privacy Act of 1974 (5 U.S.C. 552a). The privacy of data records will be explained to all respondents during the consent process and in the consent forms.
The Project AWARE-SEA national evaluation is not collecting any individual-level, primary data of a sensitive nature from students, teachers, program participants, or other entities participating in the national evaluation. The national evaluation will conduct secondary data analysis of potentially sensitive data collected through existing school information systems and student- and teacher-level surveys. These data will be provided to the national evaluation team in a de-identified and aggregated manner, so that no individual-level linkages can be made between student survey data and individual respondents.
The estimated number of respondents, responses per respondent, and burden hours are described for each NITT-Project AWARE-SEA data collection activity below. There are no direct costs to respondents other than their time to complete the instrument. Table 4 provides annualized information on the number of respondents, number of responses, burden, and cost details for each instrument. Respondent costs are calculated as total annual burden hours times the average hourly wage for associated job categories (as reported in the 2014 Occupational Employment Statistics [OES] by the Bureau of Labor Statistics [BLS]). The test procedures used to determine burden hours are detailed in Section B.4.
Telephone and In-person Questionnaires to support review and update of the AWARE-SEA Activities Inventory. Annual questionnaires will be conducted with an identified point of contact in each SEA (the Project Director or Project Coordinator), each LEA (the LEA Coordinator), and each participating school (either the school principal, school coordinator or other identified point of contact most knowledgeable about AWARE-SEA-related activities conducted in the school). Each questionnaire will be conducted annually to review and update the AWARE-SEA Activities Inventory with 20 SEA-level respondents, 60 LEA-level respondents, and approximately 432 school-level respondents. The SEA- and LEA-level respondents represent a census, while the school-level respondents include a stratified, random sample of schools participating in the AWARE-SEA initiative and identified comparison schools in the 10 substudy SEA grantees. The questionnaire will be administered on an annual basis beginning in July 2016, and take approximately 1 hour to complete at each administration.
SEA Collaborative Partner Survey. The SEA-CPS will be administered electronically through a Web-based data collection portal to all identified collaborative partners on the State Management Team (SMT). The survey respondents will include, at a minimum, the SEA project director and SEA project coordinator; project coordinators from each of the partner LEAs; youth and family representatives; and partners representing state mental/behavioral health, state criminal/juvenile justice agencies, and other identified partner agencies at the state level, such as state Medicaid directors and child welfare, early childhood, and faith-based organizations. There will be approximately 24 collaborative partner respondents at each SEA grantee who will complete the SEA-CPS. The survey will be administered on an annual basis beginning in July 2016, and take approximately 30 minutes to complete at each administration.
LEA Collaborative Partner Survey. The LEA-CPS will be administered to identified representatives from partner agencies in each LEA. The survey respondents will include key stakeholders from local mental health agencies or service providers, participating schools, and the LEA, as well as the LEA coordinator, school coordinators or principals from prioritized AWARE-SEA schools, and family and youth perspective representatives. The LEA-CPS respondents will be identified in collaboration with the SEA project director and LEA project coordinator, and administrated electronically through the Web-based data collection portal. The survey will be administered twice during the grant period, in July 2016 and July 2019, with 15 respondents in each of the 62 LEAs.1 The survey will take approximately 30 minutes to complete.
Teacher Mental Health Literacy Survey. A random sample of 24 teachers will be selected from each of the schools selected to participate in the school-level coordinator questionnaire data collection (432 schools, see above). The teachers will complete the Mental Health Literacy twice during the grant period, in October 2016 and February 2019, via the Web-based data collection portal. The survey will take approximately 30 minutes to complete.
Collaborative Partner Interviews. Approximately 160 core staff (8 SEA-level collaborative partners in each SEA grantee) are expected to participate interviews, to be conducted annually beginning in mid-2016 and continuing through mid-2019. The interviews will be conducted in-person during site visits in 2016 and 2019, and over the phone in 2017 and 2018. Each interview will take approximately 1 hour to complete per response. The estimated burden time is based on the evaluation team’s expectations of the time required to conduct these interviews; the interviewee will not need to look up or compile any information for this interview. Actual interview lengths may vary, but will not exceed the burden estimate.
Student Focus Groups. The Project AWARE-SEA national evaluation will conduct student focus groups during site visits during the 2016-17 school year and 2018-19 school year. The focus groups will be conducted with approximately 8-10 students in each of four schools in one LEA associated with each SEA grantee for a total of 1,000 students participating in focus groups at each of the two site visits. Each focus group will last approximately 60 minutes.
School Information Systems Data Transfer Protocol. School-level information will be provided to the AWARE-SEA national evaluation from the stratified random sample of schools participating in AWARE activities and the selected comparison schools (N = 432). Initial discussions with each LEA community indicate that most school information systems are at the LEA-level, but burden is estimated at the school level to account for instances where schools have their own unique information system. Each abstraction/data upload will take approximately 90 minutes, on average, with more time required in the first abstraction as data providers become familiar with the protocol and go over any questions with the national evaluation team. These data will be abstracted annually beginning in July 2016, and will be collected retrospectively and prospectively to cover five school years: 2014-2015 through 2018-2019.
Teacher School Climate and School Safety Survey Data Transfer Protocol. Data from existing teacher and student surveys in selected schools (N = 432) participating in the national evaluation will also be provided to the national evaluation on an annual basis. These data are already available at the school- and LEA-level, so the burden estimate is based on becoming familiar with the protocol for transferring data to the national evaluation team, and then submitting those data as they are available. The student survey data abstraction/ data upload and the teacher survey data abstraction/ upload will each take 90 minutes, on average, with more time required in the first abstraction as data providers become familiar with the protocol and go over any questions with the national evaluation team. These data will be abstracted annually beginning in July 2016, and will be collected retrospectively and prospectively to cover five school years: 2014-2015 through 2018-2019.
Table 4. Annualized Data Collection Burden
Instrument |
Number of Respondents |
Responses per Respondent |
Total Number of Responses |
Hours per Response |
Total Burden Hours |
Average Hourly Wage |
Total Respondent Costa |
SEA Leadership Questionnaire |
20 |
1 |
20 |
1 |
20 |
$44.23 |
$885 |
LEA Coordinator Questionnaire |
62 |
1 |
62 |
1 |
62 |
$44.23 |
$2,742 |
School Coordinator Questionnaire |
432 |
1 |
432 |
1 |
432 |
$44.23 |
$19,107 |
SEA-Collaborative Partner Survey |
480 |
1 |
480 |
.5 |
240 |
$22.71 |
$5,450 |
LEA-Collaborative Partner Survey |
930 |
1 |
930 |
.5 |
465 |
$22.71 |
$10,560 |
Collaborative Partner Interviews |
160 |
1 |
160 |
1 |
160 |
$22.71 |
$3,634 |
Teacher Mental Health Literacy Survey |
10,368 |
1 |
10,368 |
.5 |
5,184 |
$27.71 |
$143,649 |
Student Focus Groups |
1,000 |
1 |
1,000 |
1.0 |
1,000 |
$7.25 |
$7,250 |
School Information Systems Data Abstraction |
125 b |
1 |
125 |
1.5 |
188 |
$26.52 |
$4,986 |
Student Survey Data Abstraction |
125 b |
1 |
125 |
1.5 |
188 |
$26.52 |
$4,986 |
Teacher School Climate and School Safety Survey Abstraction |
125 b |
1 |
125 |
1.5 |
188 |
$26.52 |
$4,986 |
TOTAL |
13,377c |
|
13,827 |
|
|
$209,685 |
a Total respondent cost is calculated as total burden hours x average hourly wage. Average hourly wages include $44.23 for education administrators; $22.71 for AWARE-SEA core staff, partners and other stakeholders; $27.71 for primary and secondary school teachers; $7.25 for students; and $26.52 for primary and secondary school support staff. Student wage is based on the federal minimum wage. All other wages are based on the May 2014 BLS OES, http://www.bls.gov/oes/current/oes_nat.htm. If mean hourly wage was not provided, it was computed assuming 2,080 hours per year.
b Secondary data abstraction will be collected at the school level for all sample schools (N = 432), but the number of respondents is calculated based on whether the school information systems and existing student and teacher surveys are consistent across SEAs and/or LEAs, or whether they vary from school to school. Based on preliminary discussions with the grantees, SAMHSA estimates that five SEA grantees will be able to provide data for all sample schools in the SEA (N = 5 SEA respondents), the data will be provided from LEAs in ten of the SEA grantees (N = 30 LEA respondents), and the remaining five SEA grantees will have school information systems and surveys that differ at the school level (N = 90 school respondents). Therefore 125 respondents will provide the secondary data that covers the 432 sample schools.
c This is an unduplicated count of total respondents.
There are no respondent costs for capital or start-up or for operation or maintenance.
The annualized cost to the government is approximately $1,123,259. The estimated 5-year total cost to the government for the data collection is $5,616,295. This includes approximately $1,111,402 per year (or $5,557,011 total) for developing the instruments; programming and maintaining the online data collection system; providing data collection training to grantees; contractor labor for managing data collection; processing, cleaning, and housing data; and analyzing and reporting data. Approximately $11,857 per year (or $59,285 total) represents SAMHSA costs to manage/administer the data collection and analysis for 10% of one employee (GS-14-4, $118,570 annual salary).
This is a new data collection.
Table 5 outlines the key time points for the study and for the collection of information.
Table 5. Time Schedule for Data Collection
Instrument |
Year 1 (2015) |
Year 2 (2016) |
Year 3 (2017) |
Year 4 (2018) |
Year 5 (2019) |
AWARE-SEA Activities Inventory / Questionnaires |
|
X |
X |
X |
X |
SEA-Collaborative Partner Survey |
|
X |
X |
X |
X |
LEA-Collaborative Partner Survey |
|
X |
|
|
X |
Teacher Mental Health Literacy Survey |
|
X |
|
|
X |
Collaborative Partner Interviews |
|
X |
X |
X |
X |
Student Focus Groups |
|
X |
|
|
X |
School Information Systems Data Abstraction |
|
X |
X |
X |
X |
Student/ Teacher Survey Data Abstraction |
|
X |
X |
X |
The NITT-Project AWARE-SEA evaluation will help SAMHSA reach its diverse stakeholders through targeted products and innovative dissemination venues. The evaluation’s objective for all reports and dissemination products is to provide user-friendly documents and presentations that help SAMHSA successfully disseminate and explain the findings. The dissemination plan includes products in a variety of formats for a variety of target audiences. Audiences for these reports will include Congress, SAMHSA Centers, the evaluation’s SAMHSA Contracting Officer’s Representatives (CORs), NITT-Project AWARE-SEA grantees and participants, and the broader mental health and substance abuse fields (e.g., academia, researchers, policy-makers, providers). The NITT-Project AWARE-SEA evaluation recognizes that different audiences are best reached by different types of report formats. For example, reports to Congress will require materials that are concise but offer policy-relevant recommendations. Reports created for SAMHSA Centers and the CORs will require more in-depth information, such as substantive background and discussion sections, to supplement the analytic approach. Reports created for NITT-Project AWARE-SEA grantees will be concise handouts with helpful and easy-to-read graphics on performance data rather than lengthy text. The NITT-Project AWARE-SEA evaluation will develop an assortment of dissemination products, including short and long analytic reports, congressional briefings, annual evaluation reports, research and policy briefs, ad hoc analytic reports, journal articles, best practice summaries, and conference or other presentations.
The process and outcome evaluation questions under each of the five priority areas will be answered through a mixed method approach that incorporates qualitative and quantitative techniques. The combination of these approaches will provide SAMHSA with rigorous findings that link Project AWARE-SEA collaborative processes, planning activities, and implementation activities to observed outcomes in mental health literacy, capacity to identify and appropriately respond to mental health needs, a comprehensive mental health service array and access, and improvements in outcome measures of school safety, school climate, and student coping and resiliency. These analyses will support findings for both Project AWARE-SEA-specific evaluation questions as well as the NITT consolidated evaluation.
Data collected from grantee document reviews, the Activities Inventory, key stakeholder interviews, and focus groups will be subjected to qualitative analyses. These analyses will help develop a profile of each SEA grantee and each of its LEA partners, including collaborative structure, the way the collaborative teams work together, the planned implementation of the program, the actual implementation activities and associated outputs, and perceptions regarding program impact on service availability, programs to address AWARE goals, and overall school safety and climate. Qualitative analyses will also be used to illustrate effective components of the AWARE-SEA initiative, and provide more depth and understanding to the quantitative findings.
A key aspect of the qualitative data analysis is determination of what the SEAs and LEAs actually implement. Qualitative measures will be analyzed through (1) a situation analysis and problem identification informed by interviews with AWARE SEA and LEA leadership and key stakeholders and (2) a revision of the AWARE-SEA inputs, activities, and outputs derived from the interviews in addition to document review and qualitative data collected during site visits. The information about outcomes and impact will initially be derived from stakeholder interviews (i.e., What are the intended outcomes and impacts of the program?), with augmentation from the data collected as part of the outcome evaluation (i.e., What are the actual outcomes and impacts?).
A number of quantitative analysis approaches will be used to examine the process and outcomes of Project AWARE-SEA collaborative, planning and implementation activities. The analytic techniques described below will explore the linkages between collaborative processes, initiative programs and activities, SEA-level outcomes, LEA-level outcomes, and school-level outcomes. Each approach is described briefly below and will be tailored to answer each identified evaluation question.
Multilevel Latent Growth Models (MLLGM). For the analyses pertaining to Project AWARE-SEA activities’ effects on key outcomes, our primary analytic framework will be the MLLGM framework, which will be constructed to assess the extent to which changes over time among a series of potential mediators account for the effects of partnership structure on outcomes, with models that are specific to school types. The basic linear MLLGM assumes a three-level structure where individuals-nested-within-grantees were measured repeatedly over time, and it incorporates predictors of changes in outcomes over time to assess which factors are related to increases in positive outcomes, decreases in negative outcomes, or both, within and across grantees.
Propensity Scoring Approaches. One of the primary challenges in the AWARE-SEA evaluation will be accounting for the nonrandom selection of treatment and comparison schools. Propensity scoring approaches balance measured covariates that influence the probability of selection into two or more non-experimental groups and also influence treatment outcomes. These scores can then be used to weight outcome analyses to produce unbiased estimates of the intervention effect. Propensity scoring will be useful for making stronger causal inferences with regard to (1) understanding and accounting for differences between intervention schools within SEAs and nonintervention schools within SEAs, (2) disentangling the effects of NITT funding from other available funding, and (3) isolating the impact of a specific intervention from that of other interventions.
Integrative Data Analysis. Within Project AWARE-SEA, there will likely be differences in measures across data sources, particularly as the national evaluation is relying on existing data from student surveys, teacher surveys that measure school climate and school safety, and school information systems. Therefore, integrative data analysis will be used to harmonize different measures of each outcome. The harmonization process involves (1) creating a common measure for questions that are worded slightly differently from each other but are comparable and (2) using response scales (e.g., Likert-type scales, ordered categories) that can be condensed to their least common denominator (e.g., ever used or never used).
Scale scores can also be adjusted for differential item functioning (i.e., item bias) across demographic factors (e.g., gender, race/ethnicity) while preserving true differences between groups on the scale score. If a difference is found among items across these factors, different item parameters will be used to score the outcome variables in a manner that still ensures scale score comparability even if that difference is present. The last step would be the scoring of all available observations across all available time points, incorporating different item parameters across demographics.
Latent class analysis/finite mixture models (LCA) will be used to cluster units (e.g., schools and LEAs) into groups on the basis of various combinations of AWARE-SEA collaborative structures and implementation activities.
Meta-regression, which uses summary data as outcome data, will be explored for evaluation questions related to SEA-level variation, which will necessarily involve small sample sizes; meta-regression uses the same multilevel analysis framework (and corresponding software) as proposed for multilevel latent growth modeling. Effect sizes will be calculated for changes over time within grantee and standard errors for the effect sizes to calculate meta-regression weights in a manner similar to standard meta-analysis models. Key predictors can then be used to account for variability in effect sizes.
OMB approval expiration dates will be displayed.
There are no exceptions to the certification statement. The certifications are included in this submission.
1 One of the SEAs has partnered with 5 LEAs, while the remaining 19 SEA grantees have each partnered with 3 LEAs.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Snaauw, Roxanne |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |