Multi-Site Evaluation of the Safe Schools Healthy Students (SS/HS) State Program
Supporting Statement
The Substance Abuse and Mental Health Services Administration’s (SAMHSA’s) Center for Mental Health Services (CMHS) is seeking Office of Management and Budget (OMB) approval for data collection associated with the multi-site evaluation (MSE) of the Safe Schools/Healthy Student (SS/HS) Initiative. The SS/HS Initiative is authorized under the Safe and Drug-Free Schools and Communities Act (20 U.S. Code [U.S.C.] 7131), Public Health Service Act (42 U.S.C. 290[hh]), and Juvenile Justice and Delinquency Prevention Act (42 U.S.C. 5614[b][4][e] and 5781 et seq). This legislation sets aside funds for states/tribes, Local Education Authorities (LEAs), and their community partners to improve collaboration across all child, youth, and family serving organizations, improve access to the availability of evidence-based prevention and wellness promotion practices, and focus on both school-based and community-wide strategies to prevent violence and promote the healthy development of children and youth.
The legislation also mandates that an evaluation of the program be carried out and reported to Congress. In fiscal year 2014, the decision was made to bring the SS/HS to scale through statewide adoption of the SS/HS State program by 7 States/Tribes and their 21 LEAs that will participate in the multisite evaluation (MSE).
Consistent with the tenets of the President’s “Now is the Time” plan, the SS/HS State program seeks to create safe and supportive schools and communities. The program also seeks to build state- and community level partnerships among educational, behavioral health, and criminal/juvenile justice systems that promote systems integration and policy change and sustainable policies, infrastructure, services, and supports.
The expectations of the statewide expansion of the SS/HS State program are to:
increase the number of children and youth who have access to behavioral health services;
decrease the number of students who abuse substances;
increase supports for early childhood development;
improve school climate; and
reduce the number of students who are exposed to violence.
Since the SS/HS program was initiated in 1999, there have been significant and relevant changes in the school and community context regarding safety and support for students. For example, between 2000 and the current time, more than 160 new state bullying laws, have been passed or revised; 49 states currently have laws concerning bullying in schools. These laws vary significantly in their provisions. However, prevention, and particularly mental health services, is relatively neglected in state bullying laws.
Research has demonstrated relationships between school climate and school performance, including attendance, graduation rates, connectedness, and academic achievement. Federal, state, and local educational agencies have also become increasingly concerned about the social, emotional, and relational climate of their schools. Research has also documented the pervasive use of out- of- school suspension and expulsion, and the clear negative impacts on school connectedness, attendance, graduation, and performance. Concerns about the predominant use of suspension and expulsion as primary responses to student behavioral problems have increased, with corresponding interest in alternatives (e.g., youth courts, restorative justice).
Section 3 of the President’s report, “Now Is the Time”, identifies initiatives for making schools safer, such as improved coordination with law enforcement, and to “increase awareness of mental health issues and connect young people with behavioral health issues and their families with needed services.” In the aftermath of the Sandy Hook Elementary School tragedy, strategies and initiatives have been proposed that are consistent with the SS/HS Framework. These strategies can benefit from findings and established practices developed through the implementation of the SS/HS State program.
The MSE is comprised of two primary components:
Planning, Collaboration, and Partnership Study
State Key Informant Interview
District Key Informant Interview
State Collaborator Survey
District Collaborator Survey
State Collaboration Indicator Data Instrument
District Collaboration Indicator Data Instrument
Implementation Study
Key Informant Interview
School-Level Survey
Workforce Study
No additional instruments will be used for this study. Data will be gathered from the Planning, Collaboration and Partnership Study and the Implementation Study.
The request is for approval of the data collection instruments to be used in collecting data for this evaluation.
The SS/HS program has been at the forefront of policy and programs to make schools safe, promote positive youth development, and provide supportive learning environments for more than 14 years. At the time of its implementation in 1999 the SS/HS program was a collaborative grant program supported by three Federal departments—the U.S. Departments of Health and Human Services, Education, and Justice. The SS/HS program embodied multiple components of school safety and support. These included the need to address multiple implications for youth development and academic achievement; the need for comprehensive and multifaceted solutions; the critical role of youth and family involvement; and the essential need for integration of historically siloed community services. Key elements of the SS/HS program, as implemented by earlier cohorts, included the following:
Sponsorship and administration of the program through a collaboration of the U.S. Departments of Education (ED), Health and Human Services (HHS), and Justice (DOJ). This collaboration addresses the long understood problem of service fragmentation, and models the need for collaboration across agencies at the point of delivery.
From the program’s inception, the requirement that SS/HS grantees establish school-focused partnerships among education, mental health, juvenile justice, and law enforcement. This requirement acknowledged the growing recognition that safe and supportive schools can be best created when schools and communities work together.
Provision of a framework to guide grantees in their comprehensive planning and collaborative activities. In 2007, this framework was revised along with other grant features (e.g., funding limits increase from 3 to 4 years), but the elements remain essentially the same. Currently, the framework includes the development of evidence-based strategies in each of the following areas:
Safe school environments and violence prevention activities
Alcohol, tobacco, and other drug prevention activities
Student behavioral, social, and emotional supports
Mental health services
Early learning programs
In addition, the framework included two components: Strategic Approaches (collaboration and partnership, technology, policy change and development, capacity building, systemic change and integration), and Guiding Principles (cultural and linguistic competency, serving vulnerable and at-risk populations, youth guided and family driven, developmentally appropriate, resource leveraging, sustainable, evidence-based interventions).
The SS/HS framework provided an important tool for grantees, particularly when a central objective was to encourage collaboration among education, behavioral health, and criminal justice partners. The framework then supported grantees in concretely designing their strategic plans, and provided more actionable strategies to collaborate with local organizations (e.g., the SS/HS Model for Mental Health Promotion, Prevention, and Treatment; catalogues of evidence-based interventions). These studies provided substantial evidence that SS/HS programs were successful in achieving intended outcomes, including reductions in school violence, youth substance abuse, and perceptions of school safety. SS/HS programs were found to produce particularly large improvements in access to and use of mental health services for youth. The evaluation has demonstrated that SS/HS programs can successfully achieve their intended outcomes (Bershad et al., 2012; Storey et al., 2012).
As a result of the above referenced evidence, the initiative is now being piloted as a state/tribal intervention designed to bring the program to scale. With this effort to promote widespread adoption of the SS/HS program, SAMHSA now seeks to build upon the lessons from this important grant program by providing funds to disseminate the lessons learned from SS/HS by engaging state/tribal and community (including local education agencies) partnerships that will result in the successful implementation of comprehensive school violence prevention initiatives that are guided by the SS/HS model. Issues that affect the learning environment of schools - such as bullying, fighting, alcohol and substance use, need for mental health services, and truancy - cannot be solved by schools alone. Collaboration allows for combined knowledge, skills, and resources of various local public, private, and community agencies to be used in responding these issues. The SS/HS mission continues to support school and community partnerships in their efforts to develop and coordinate integrated systems that create safe, drug-free, and respectful environments for learning and to promote the behavioral health of children and youth.
The SS/HS evaluation conducted under the authority of the Secretary of Health and Human Services is mandated under 42 U.S.C. 290(bb), item (f):
The Secretary shall conduct an evaluation of each project carried out under this section and shall disseminate the results of such evaluations to appropriate public and private entities.
Process evaluations are an often-overlooked yet essential component of health promotion interventions. Additionally, there is presently a dearth of knowledge about the factors responsible for the successful scaling up of prevention programs. The goal of the MSE, therefore, is to assess the extent to which this pilot effort to facilitate wide-scale adoption and operation of the SS/HS State program succeeds in building state- and community level partnerships among educational, behavioral health, and criminal/juvenile justice systems that promote systems integration and policy change and sustainable policies, infrastructure, services, and supports.
The MSE will use its findings to develop criteria for evidence of implementation which will build our capacity to describe and assess the overall program. In addition, SS/HS grantees will receive appropriate guidance from experts in evaluation, and methodology. Further, the evaluation will be used to inform SAMHSA’s policy-making decisions by identifying and describing successful approaches to coordination among multiple service systems, assessing grant program performance, and accurately and comprehensively describing the intervention. The lessons learned from this evaluation will inform policymakers in developing policies and programs that enhance the safety of the nation’s schools and increase children’s access to mental health services. Because of the lack of information about bringing demonstration projects to scale, the findings of the MSE will be invaluable in contributing to the wider understanding of how to successfully bring this and future initiatives to scale.
The evaluation will serve to:
assess the extent to which implementation of comprehensive school violence prevention initiatives, guided by the SS/HS Framework, is achieved at both the state/tribal and community levels;
determine the breadth and volume of activities necessary to achieve coordination across multiple service systems;
identify and describe the elements or activities that are associated with improved child wellness; and
estimate the extent to which states/tribes and communities improve access to mental health services for target populations and reduce subpopulation disparities in access, services, and outcomes.
Clearance Request
This submission is to request OMB clearance for data collection components for 3 years (years 2-4 of grantee funding) of the multi-site evaluation of the SS/HS program. The evaluation will use two primary study components—the Planning, Collaboration and Partnership (PCP) Study, and the Implementation Study—to investigate the three critical areas of inquiry:
1) Examine the degree to which the SS/HS Framework has been successfully implemented;
2) Assess grant program performance; and
3) Accurately and comprehensively describe the intervention.
Both study components will incorporate assessment of workforce domains, as part of a Workforce Study, to understand and describe the elements of the mental health workforce that facilitated or functioned as barriers to SS/HS State program planning and implementation.
A detailed description of the multisite evaluation goals, questions, and data collection instruments by study is presented below.
Planning, Collaboration and Partnership Study: The PCP study will assess the level of collaboration among SS/HS partners, the barriers/facilitators to interagency collaboration and the system-level efforts to address racial and ethnic minority health disparities as outlined in Table 1. Data will be collected by means of key informant interviews, web-based surveys, and document reviews.
Table 1: Planning, Collaboration, and Partnership Study Research Questions and Objectives
Data Collection Instrument |
Objectives |
Research Questions |
State Key Informant Interviews (KIIs) (Attachment A) |
Understand the context and the factors that impact the implementation of the SS/HS Initiative. |
What factors facilitate/hinder widespread adoption and sustainability of the SS/HS Framework values, principles, and practices? |
District KIIs (Attachment B) |
||
State Collaborator Survey (Attachment C) |
Determine the context of interagency collaboration, adherence of planning activities to the Framework, system-level efforts to address racial and ethnic minority health disparities and the level of collaboration among partners. |
What were the barriers/facilitators to interagency collaboration, partnership, development, and shared decision-making? How were they addressed? Did the comprehensive plan accurately reflect the components of the SS/HS Framework? What factors facilitate/hinder the development of the comprehensive plan? What roles were played by consumers or those with lived experiences in the planning process? How did this facilitate the planning process? What role do systems-level factors—such as planning, collaboration, and partnership—play in promoting or prohibiting the effectiveness of strategies and practices aimed at reducing racial and ethnic minority health disparities? What is the level of interaction and collaboration among partners? |
District Collaborator Survey (Attachment D) |
||
State Collaboration Indicator Data Instrument (Attachment E) |
Understand the context in which collaboration occurs at state/tribal, LEA, and school levels. |
What are the structure, processes, and activities of the collaboration network at state/tribal, LEA and school levels? |
District Collaboration Indicator Data Instrument (Attachment F) |
Implementation Study: The implementation study will be used to determine whether services and supports were delivered in keeping with the SS/HS Framework; ascertain the depth and volume of these activities required to achieve coordination across the multiple services; and to examine the extent to which states/tribes and communities improve access to mental health services.
The instruments listed in Table 2, will provide an understanding of the relationship that exists between the program context (i.e., setting characteristics) and program processes (i.e., levels of implementation).
Table 2: Implementation Study Research Questions and Objectives
Data Collection Activity |
Objectives |
Research Questions |
KIIs (Attachment G) |
Assess the services and supports that were implemented, the ability of grantees to meet program goals and the policies that promoted or hindered program implementation
|
What services and supports were developed and implemented? How were behavioral health disparities in access, services, and outcomes across subpopulations addressed? What are the barriers/facilitators to implementing the comprehensive plan? What factors facilitate/hinder widespread adoption and sustainability of the SS/HS Framework (values, principles, and practices)? What policies at the state/tribal and/or community level facilitate/hinder implementation? To what extent were grantees able to implement systems approaches to addressing behavioral health disparities? |
School-Level Survey (Attachment H.1) |
Assess whether there is adherence to SS/HS Framework |
To what extent are the three components—SS/HS elements, guiding principles, and strategic approaches—of the SS/HS Framework implemented? |
Workforce Study: The workforce study will identify and describe activities such as trainings in health disparities awareness that are associated with improved child wellness. The research questions to be answered by this study are listed in Table 3. However, the data for the workforce study will be gathered from the interviews completed as part of the Planning, Collaboration and Participation and Implementation studies and thus, will not require separate data collection activities.
Table 3: Workforce Study Research Questions and Objectives
Data Collection Activity |
Objectives |
Research Questions |
Key Informant Interviews |
Establish the types of mental health workforce training has occurred and whether these trainings have increased awareness of health disparities and improved the cultural competency of the workforce |
What developments with respect to the mental health workforce facilitated/hindered implementation? What training programs for the mental health workforce were initiated as a result of implementation of the comprehensive plan? To what extent does the mental health workforce demonstrate increased awareness of health disparities and improved cultural competency? |
What follows is a description of the major components of the MSE, their associated data collection instruments, and the uses of the information collected.
Multi-site Evaluation Design and Data Collection Instruments
The MSE will incorporate qualitative and quantitative methods to collect comprehensive information about the wide-scale adoption of the SS/HS State program. The qualitative methods used in the Key Informant Interviews (KIIs) will assess the extent to which implementation of comprehensive school violence prevention initiatives, guided by the SS/HS Framework, is achieved at both the state/tribal and community levels; determine the breadth and volume of activities necessary to achieve coordination across multiple service systems; identify and describe the elements and activities that are associated with improved child wellness; and estimate the extent to which states and communities improve access to mental health services for target populations and reduce subpopulation disparities in access, services, and outcomes.
Quantitative methods will be utilized to gather information through web-based surveys that will evaluate program adherence to the SS/HS Framework values, principles, and practices, dose of services delivered, reach, recruitment, and context. The various components of the MSE studies and data collection strategies are described below.
Planning, Collaboration, and Partnership Study
State Key Informant Interview: The state KII guide (see attachment A) will be used to conduct interviews annually during Years 1, 2 and 3 of the evaluation regarding interagency planning and collaboration and adherence to the SS/HS Framework. Topics will include the service model, partnerships and interagency collaboration, program implementation adherence, plan deviations, and state/tribal and local policy development. Responses will be compared over time to assess positive development of the program model, emerging barriers and facilitators to implementation, and evolving solutions. Key project staff (e.g., project coordinators/co-coordinators); informed representatives of the state management team (SMT) member agencies and organizations, representatives of other agencies/organizations collaborating at the state level will be recruited for interviews.
District Key Informant Interview: The district KII guide (see attachment B) will identify, through the staff’s own descriptions of activities, the degree to which critical SS/HS Framework elements are operationalized, as well as the degree to which principles and strategies are acknowledged and integrated as part of the service processes. Topics include the provider’s approach to service provision (sensitivity to disparities, culturally competent), the coordination of services across the LEA and other local agencies, local policy and protocol development, and barriers/facilitators at the local level that influence the adoption, integration, and sustainability of SS/HS principles. Again, responses will be compared over time to assess positive development of the program model.
State Collaborator Survey and District Collaborator Survey: The State and District Collaborator Surveys (see attachments C & D) are web-based tools that will help to understand the collaboration and partnership aspects of this evaluation at the state and district levels. Surveys will include items from previously developed surveys including the SS/HS Partnership Inventory Merrill et al. (2012) and Inter-professional Team Collaboration for Expended School Mental Health (IITC-SEMH) (Mellin et al., 2010). The surveys will therefore assess respondents’ perceptions of how local SS/HS implementation partnerships function in terms of partner goals, resources, culture and values, and roles and responsibilities, as well as leadership and collaboration among partners. The surveys will also identify elements of inter-professional collaboration among entities working toward expanded school mental health. Content areas measured will include reflection on process, professional flexibility, newly created professional responsibilities, and role interdependence. The surveys will include questions that rate the performance of coalitions and the information shared.
State Collaboration Indicator Data Instrument and District Collaboration Indicator Data Instrument: The State and District Collaboration Indicator Data Collection Instruments (see attachments E & F) will be administered on a quarterly basis to learn about the number of meetings and attendance, trainings and attendance, information resources provided, number of assistance requests, and resource leveraging on each level.
Implementation Study
Key Informant Interview: The KIIs (see attachment G) that will take place as part of the Implementation study will be conducted with persons involved with the program at State and district levels to understand the elements necessary for successful implementation of the program as well as challenges to implementation. Interviews will also include questions to learn about the training that the mental health workforce received as a result of the implementation of the program.
School-Level Survey: The School-Level Survey (see attachment H.1) will collect data regarding the implementation of the SS/HS program at the local level. Survey items will include questions from the Evidence-Based Practice Attitude Scale (EBPAS) (Aarons, 2004); Mental Health Service Integration Survey (MHSIS) (Burton, Massey, & Lucio, 2012); and the School Mental Health Quality Assessment Questionnaire (SMHQAQ) (Weist, Sander, Walrath, et al. 2005).
The EBPAS assesses mental health and social service provider attitudes toward adopting evidence-based practices
The MHSIS will address the following evaluation questions: (a) barriers and facilitators to implementing the comprehensive SMH service plan and (b) factors associated with successful adoption and sustainability of SS/HS values and principles
The SMHQAQ is a tool focused on the integration of school mental health services delivered in schools.
SMHCI is designed to look at the capacity of schools to address the mental health needs of students. The schools can be rated along a continuum using the three individual subscales of intervention, early recognition & referral, or prevention & promotion
Workforce Study
Key Informant Interview: Questions regarding the training and workforce development opportunities that were available and/or lacking throughout the program will be added to the KIIs that will occur as part of the PCP and implementation studies. Separate interviews will not be conducted as part of this study.
Uses of Information Collected through the Multi-site Evaluation
Data collected as part of the cross-community evaluation will be useful to SAMHSA and its partners; other Federal agencies; legislators; federal administrators; the fields of bullying, violence, and substance abuse prevention; individual youth and their families; and the communities in which they live. Comprehensive information gathered from multiple communities at various levels and stages of their programmatic activity will augment the existing knowledge base. Evaluation findings will also serve to inform the future implementation of programs implemented in response to federal initiatives such as those outlined in the President’s “Now is the time” plan.
If these data are not collected, policymakers and program planners at the Federal and local levels will not have the necessary information to determine if the SS/HS grantees are using the framework to work collaboratively across sectors and whether or not they are meeting their objectives.
Efforts will be made to limit burden on individual respondents who participate in the MSE through the use of technology. Data collection instruments will be administered via the Web and telephone. Below is a description of the Web-based data collection and management system will be used for data collection.
Web-based data collection and management system
Web-based surveys and forms will be used for the following data collection activities:
State Collaborator Survey (PCP)
District Collaborator Survey (PCP)
State Collaboration Indicator Data Instrument (PCP)
District Collaboration Indicator Data Instrument (PCP)
School-Level Survey (Implementation)
Data will be collected by means of web-based surveys through the SS/HS Evaluation Data System (SHEDS). The SHEDS is a web-based data repository that the MSE team will develop to accommodate all required data collection, management, and dissemination needs of the SS/HS evaluation. The SHEDS will allow for data collection from various methods, including direct administration of Web-based surveys to SEA and LEA program staff, as well as LEA local community members and direct entry of data that tracks the program activities using the Indicator data entry form. The SHEDS will also serve as an integrated information center where information about the program will be accessible to program partners.
Use of Web-based surveys is anticipated to decrease respondent burden compared to alternative methods such as a paper format, by allowing for direct transmission of the instrument to and from survey respondents. In addition, the data entry and quality control mechanisms built into the Web-based system will reduce errors that might otherwise require follow-up, thus reducing burden compared to a hardcopy administration. Respondents will also be able to complete the survey at a time and location that is convenient for them. The web-surveys associated with the MSE will recruit respondents to participate through an e-mail invitation that includes the Web-site URL to complete the survey, which will further increase the ease of responding.
Additional efforts to lessen study burden on participating states/tribes and communities will include the strategic use of secondary data such as the documents and archival records related to the demographics of children and youth participating in the program.
a. Existing Research
According to a report by the U.S. Department of Health and Human Services, 20% of U.S. children have a diagnosable mental disorder, with 5–9% classified as seriously emotionally disturbed (HHS, 1999). These children and their families face a host of barriers that seriously restrict access to community-based services (Owens et al., 2002). Recognizing the potential that school systems have in addressing this problem, legislation enacted over the past decade (NCLB of 2001 and IDEA of 2004) emphasized that schools need to support behavioral development, particularly for children with identified mental health problems (Atkins et al., 1998; Walter et al, 2011). Schools are convenient, accessible, and structurally equipped to serve children and, next to families, schools arguably hold the most appreciable influence over children (Atkins et al., 1998).
While research in school mental health has contributed significantly to our understanding of what makes services effective (Atkins et al., 1998; Rones & Hoagwood, 2000; Kutash,
Duchnowski, & Green, 2011), implementation challenges persist (Atkins, Hoagwood, Kutash, & Seidman, 2010). For example, not much is known about how to ensure the successful wide-scale adoption of programs such as the SS/HS State program. School mental health services are often fragmented and marginalized (Taylor & Adelman, 2000), poorly integrated into school environments (Weist, Sander, & Walrath, 2005), and are seen as hard to fit into the existing school structure (Burton, Hanson, Levin, & Massey, 2012). Serious programmatic challenges to the integration of school-based mental health services also remain because of the inherent organizational differences between schools and traditional mental health services (Adelman & Taylor, 2006). Multiple studies document that poor integration impedes implementation and outcomes (Massey, Armstrong, Boroughs, Henson, & McCash, 2005; Penuel, Riel, Krause, & Frank, 2012). The effectiveness of programs may also be compromised if the services are not sensitive to the culture and needs of youth (Yampolskaya, Massey, & Greenbaum, 2006).
Medical, behavioral health, and mental health researchers and practitioners have come to recognize the critical importance of the use of service interventions that have established evidence of their efficacy (Sacket et al., 1996; Hoagwood, Burns, Kiser, Ringeisen, & Schoenwald, 2001). These practices have been labeled Evidence-based Practices and were often promulgated with the expectation that services of proven efficacy would be easily and readily adopted in the field to improve the quality of outcomes for service recipients. Unfortunately, it is now recognized that programs are not readily adopted, and that there are significant gaps in the translation of evidenced-based best practices into workable programs in the field (Gonzales, Ringeisen, & Chambers, 2002; Proctor et al., 2007; Urban & Trochin, 2009). The simple provision of an innovation is not sufficient to ensure that it is implemented or implemented with fidelity (Wandersman et al., 2008).
Implementation requires an active effort to embed an innovation in a new organization (Aarons,2006; Greenhalgh et al., 2004; Prochaska et al., 2001) and rests with organizational, cultural, human resource, administrative issues, staff attitudes, and readiness for change (Aarons, 2004; Green, 2008; Lehman, Greener, & Simpson, 2002). Adherence to system models involves attention to measuring and maintaining the critical elements of a program practice as it is brought into the community setting (Bruns, 2008; Co-Occurring Center for Excellence, 2006). In mental health research, such adherence includes efforts to ensure that both the philosophy and principles of an intervention are adopted and supported by users at the site, and that specific behavioral, service, training, and administrative activities are appropriately mirrored in practice. The multi-site evaluation of SS/HS will present a unique opportunity to collect information from multiple grantees to assess the factors that facilitate and/or hinder wide-scale adoption and operation of the SS/HS Initiative and build state-and community level partnerships among education, behavioral health, and criminal/juvenile justice systems that promote systems integration and policy change and sustainable policies, infrastructure, services and supports. The data to be collected does not include data collected in previous studies.
The majority of the data for this evaluation will be collected from state/tribal administrators, school staff, and partners affiliated with SS/HS State program. Some of the data for this evaluation will be collected from individuals involved with public agencies, such as education, juvenile justice, and tribal entities. While respondents most likely are employed by public agencies, it is possible that some may also be employed by small businesses or other small entities. But these data collection activities will not have a significant impact on these agencies or organizations.
Table 4 shows the frequency of which the various data collection activities will take place.
Table 4. Data Collection Frequency
Data Collection Activity |
Administration Frequency |
Partnership, Planning, and Collaboration Study |
|
State Key Informant Interviews (KIIs) |
Annually years 1-3 of data collection |
District KIIs |
Annually years 1-3 of data collection |
State Collaborator Survey |
Annually years 1-3 of data collection |
District Collaborator Survey |
Annually years 1-3 of data collection |
State Collaboration Indicator Data Instrument |
Quarterly years 1-3 of data collection |
District Collaboration Indicator Data Instrument |
Quarterly years 1-3 of data collection |
Implementation Study |
|
KIIs |
Annually years 1-3 of data collection |
School-Level Survey |
Annually years 1-3 of data collection |
The collection of information for the majority of the MSE activities is scheduled to occur on an annual basis during years 1-3 of the evaluation. Annual data collection is required by the MSE because less frequent data collection will hamper the evaluation of the implementation of ongoing changes in policy on the wide-scale adoption of the SS/HS program. Without this information, it will not be possible for SAMHSA to learn about the practices that promote program sustainability. Such lessons will be important for successful implementation of programs such as the President’s “Now is the Time” plan.
Grantees will be required to complete the State and District Collaboration Indicator Data form on a quarterly basis over the duration of the grant period. Collecting this information quarterly is necessary to effectively track progress of meetings, trainings, and activities that facilitate partnership and planning needed to meet program goals. The consequences of collecting those data less frequently are the potential of losing information related to the process of developing partnerships as well as losing the ability to track the growth of the collaborative efforts of grantees over time.
The data collection fully complies with the requirements of 5 CFR 1320.5(d) (2).
a. Federal Register Notice
SAMHSA published a notice in the Federal Register, on July 18, 2014 (79 FR 42026), soliciting public comment on this evaluation. No comments were received on this data collection.
b. Consultation Outside the Agency
Consultation on the design, instrumentation, and statistical aspects of the evaluation has occurred with individuals outside of SAMHSA. The MSE team has identified an external group of relevant experts to provide guidance on the process evaluation as members of the Evaluation Advisory Panel (EAP). The EAP has expertise in school mental health, children’s mental health, substance abuse prevention, system-level program implementation, and violence prevention. They will convene to provide feedback on implementation measures; findings and dissemination activities; implications for larger-scale adoption and implementation of the SS/HS framework; and considerations and planning for subsequent evaluation of the SS/HS Framework. Consultation with the EAP will begin in April 2014 and will continue as needed throughout the grant-funding period.
For each of the Web surveys and for key informant interviews, respondents will be selected on the basis of their roles in the SS/HS State program at the state/tribe or community, thus the respondents are known to the evaluation. Participants in the key informant interviews will be asked to use only their first names during the interviews and their names will be redacted from the transcripts prior to analyzing the data. However, specific names and contact information will be stored separately from survey responses, and individual respondents will not be identified in reports or in data submitted to SAMHSA.
The MSE team will ensure that personal identifying information is maintained on a secure, password-protected and encrypted server. All data are the property of SAMHSA and will be securely transmitted to SAMHSA at the conclusion of the project. Once data have been transferred to SAMHSA, the MSE team will destroy the data. Hard-copy data such as transcripts will be shredded and electronic data will be deleted from our servers. To protect the privacy of study respondents, aggregate reporting will be limited so that data will be reported only for cases in which there are more than 10 respondents.
The MSE team will be responsible for securing initial and annual IRB approval through ICF’s federally registered IRB (FWA- 00000845) as the IRB of record. In addition, the USF team will be responsible to its IRB for the work completed on the implementation component of the study. All data collected will be kept private to the extent determined by the laws applicable in each state.
An active consent process will be implemented that informs participants of the purpose of the evaluation, describes what participation entails, and addresses maintenance of privacy and mandated reporting. Verbal consent will be obtained for all key informant interviews. All respondents who complete web-based surveys will be required to complete an electronic consent form prior to beginning the completion of the survey.
The MSE team will store the names and contact information of respondents separately from the transcript of interviews and recordings, with a code key linking the two. Only team members involved in data collection will have access to the code keys, which will be destroyed as soon as data collection is complete. Standard procedures include limiting access to identifying information, using locked files to store completed hard-copy tools; assigning unique code numbers to participants; and following minimal data requirements when reporting findings.
Respondents will not be asked any questions of a personally sensitive nature. The subject matter of the interview and survey questions will be limited to the perceptions of grant planning and implementation activities among key stakeholders of the grants and to school employees’ perception of student behavior, substance use, violence, safety, and access to mental health services.
Table 5 below shows the burden associated with MSE data collection activities and the associated costs. The number of grantees for which burden is calculated is for 7 state/tribal grantees and 21 LEA grantees depending on the level the instrument applies to. Data collection for the MSE will cover a 3-year project period.
Table 5. Estimated Annual Burden Hours and Costs
Note: Total burden is annualized over the 3-year clearance period.
Type of Respondents |
Instrument |
Number of Respondents |
Responses per Respondent |
Total Number of Responses |
Average Hours per Respondent |
Total Annual Hour Burden |
Hourly Wage Rate |
Total Cost ($)
|
Planning, Collaboration & Partnership Study |
||||||||
Key project staff at state level (e.g., project coordinators, evaluators), SMT members |
State KIIs |
14 |
1 |
14 |
1 |
14 |
21.78 |
305 |
Key project staff at LEA level (e.g., project coordinators, evaluators), CMT members |
District KIIs |
63 |
1 |
63 |
1 |
63 |
21.78 |
1,372 |
Key project staff at state level (e.g., project coordinators, evaluators), SMT members |
State Collaborator Survey |
208 |
1 |
208 |
.5 |
104 |
21.78 |
2,265 |
Key project staff at LEA level (e.g., project coordinators, evaluators), CMT members |
District Collaborator Survey |
624 |
1 |
624 |
.33 |
206 |
21.78 |
4,487 |
Project Evaluator |
State Collaboration Indicator Data Instrument |
7 |
4 |
28 |
1.5 |
42 |
36.80 |
1,546 |
Project Evaluator |
District Collaboration Indicator Data Instrument |
21 |
4 |
84 |
1.5 |
126 |
36.80 |
4,637 |
Implementation Study |
||||||||
Program and school staff working at the state & district level |
KIIs |
56 |
1 |
56 |
1 |
56 |
21.78 |
1,220 |
Program and school staff working at the school level |
School-Level Survey
|
2,100 * |
1 |
2,100 |
.45 |
945 |
21.78 |
20,582 |
Total |
3,093 |
-- |
3,177 |
-- |
1,556 |
|
36,414 |
|
National Compensation Survey: Occupational Wages in the United States (2012, May). US Bureau of Labor Statistics (BLS) US Dept. of Labor. The category Social Scientists and Related Workers under Life, Physical and Social Science Occupations was used as an approximation for Project Evaluators. http://www.bls.gov/oes/current/oes_nat.htm#19-0000
National Compensation Survey: Occupational Wages in the United States (2012, May). US Bureau of Labor Statistics (BLS) US Dept. of Labor. The category Child, Family and School Social Workers under Life, Physical and Social Science Occupations was used as an approximation.
Link: http://www.bls.gov/oes/current/oes_nat.htm#19-0000 *10 respondents will participate in up to 10 schools in each of the 21 LEAs
There are neither start-up nor capital costs associated with data collection for respondents. There will be some additional burden on record keepers to provide potential respondent lists for data collection activities. However, these operation costs will be minimal.
Each grantee has been funded, as part of the overall cooperative agreement award, to participate in the MSE, with up to 10% of the grant award available for evaluation efforts and data collection. Therefore, no cost burden is imposed on the grantee by this information collection effort.
SAMHSA has contracted with ICF for designing and implementing the MSE under a contract totaling $1,734,686 over a 4-year period. Included in these costs are the expenses related to the following activities: development of the design and instrument package, supporting the MSE data collection processes. In addition, these funds will support the development of the Web-based data collection and management system and fund MSE staff support for data collection.
Each grantee is expected to fund an evaluator to conduct the local evaluation and to satisfy the MSE, GPRA, and TRAC requirements. Additionally, it is estimated that participating in the MSE will require 0.20 full-time equivalent (FTE) to collect information, enter information into the Web-based data collection and management system, and to conduct analyses at the local level. Assuming:
1) an average annual salary of $79,560 (BLS, 2012) for a 0.20 FTE evaluator;
2) 7 State grantees the annual cost for the multi-site evaluation at the grantee level is estimated at $111,384 annually.
It is estimated that SAMHSA will allocate 0.25 of a full-time equivalent each year for Government oversight of the evaluation. Assuming an annual salary of $100,624, these Government costs will be $25,156 per year.
The annualized cost to the government will be $570,211.
The time schedule for implementing the multi-site evaluation is summarized in Table 6. A 3-year clearance is requested for this project.
Table 6. Time Schedule
Begin data collection for 7 grantees (7 State/Tribal grantees) |
February 2015 (1 month after OMB approval estimated to occur in January 2015) |
Data collection completed for the grantees funded in FY2014 |
June 2017 |
Submit Annual Reports |
October 2014, 2015, and 2016 |
Final Report |
October 2017 |
SAMHSA requires annual reports summarizing the results of the evaluation. The MSE team will analyze data collected and prepare interim annual reports to summarize key findings. A final report on the results of the evaluation is also required and will be produced by MSE team within 30 days of the end of the contract.
Coordination among the MSE team and our evaluation partners, SAMHSA, other Federal programs and contractors, consumers, and the EAP is essential to the development and implementation of the evaluation of the SS/HS State program and subsequent sharing of evaluation findings. This includes coordinating ad hoc presentations with key stakeholders and audiences.
The analytic strategies for the process evaluation are iterative and require integrating both qualitative and quantitative evidence across grantees and years. Qualitative analyses will include content analysis of documents as well as responses to semistructured interview questions. Quantitative analysis will include descriptive and bivariate analysis of survey data as well as descriptive analysis of the Framework Implementation Inventory data. Integration of qualitative and quantitative data sources will be required to conduct detailed analysis for the PCP, Implementation, and the Workforce Studies.
Qualitative Data Analysis
To determine the level of planning, collaboration and partnership that occurs as part of the process of implementing the program, qualitative data gathered through key informant interviews will be analyzed. Interviewers will analyze the content of each interview to determine the level of planning, collaboration and partnership that is ongoing and the contexts in which such relationships are developed. They will then write summaries of each interview that clearly describe how these activities occur and the extent to which this is the case through within-case and cross-case comparisons. As the evaluation team begins to generate conclusions about the data, the team will verify these more general analyses and validate them by cross-checking and revisiting the data.
Comparisons across time and across sites will be conducted to identify consistent patterns of barriers and facilitators to implementation and integration of services, and to identify the changing status of services throughout the life of the grant. The project is supported by standards for triangulation of data (Denzin, 1978) that ensure that information from one source, such as interviews, is compared with other sources, such as documents and survey data. Triangulating results from multiple sources creates more credible evaluation results and is considered critical to the validity and reliability of findings (LeCompte & Schensul, 1999), and the use of multiple investigators/reviewers for qualitative analysis.
Social Network Analysis
The MSE staff will analyze the information collected as part of the Collaborator Survey using a Social Network Analysis (Brandes & Pich, 2011). The analysis will assess agency/organization collaborations that support school safety, student access to mental health services, bullying and violence prevention, and substance abuse prevention, as well as collaboration and partnership around SS/HS Framework components—Five Elements, Guiding Principles, and Strategic Approaches. The reports will include results which depict SEA, LEA, and local community agency relationships in the following domains: administration and decision making; development of service infrastructures; implementation of outreach activities; knowledge, adherence, and implementation of the Five Elements, Guiding Principles, and Strategic Approaches; and prevention strategy coordination. The final report will include narrative text discussing the key relationships, collaboration partnerships, and overall network cohesion, as well as barriers and facilitators to collaboration. The reports will also include sociograms—graphical depictions of connections between respondent organizations related to collaboration and partnerships. These pictographic representations help to illustrate the relationships among SEA, LEA, and local community agencies, as well as the structure of the network as a whole.
Quantitative Data Analysis
To determine the characteristics of children and youth served, the MSE team will conduct descriptive analyses that provide information about the characteristics of children and youth. Staff will review grantee health disparity impact statements and work with grantees to identify appropriate strategies for collecting data that will enable us to describe the characteristics of students.
To determine the role that systems-level factors—such as planning, collaboration, and partnership — play in promoting or hindering the effectiveness of strategies and practices aimed at reducing racial and ethnic minority health disparities, process data gathered through the annual implementation inventories will be used as covariates in our impact models and in the conduct of subgroup analyses. At the state/tribal level, questions about the activities required to successfully implement the SS/HS program will be addressed. This will be done by examining the extent to which the presence or absence of activities for each of the elements, strategic approaches, and guiding principles are present within the plan, their level of implementation, and the fidelity with which they are implemented.
We propose examining the relationships between program implementation and adherence to the framework and associated key outcomes. We will use both quantitative data gathered through an annual inventory of SS/HS Framework adherence and implementation, as well as data gathered through document reviews and key informant interviews to create variables that will be used in statistical models to examine the extent to which the process and implementation of planning, collaboration, and service delivery among grantees are associated with key outcomes. The annual inventory will include data gathered from the Evidence-Based Practice Attitude Scale (EBPAS), Mental Health Service Integration Survey (MHSIS), and the School Mental Health Quality Assessment Questionnaire (SMHQAQ), School Mental Health Capacity Instrument (SMHCI) instruments that tell us the extent to which the SS/HS elements, guiding principles, and strategic approaches of the SS/HS Framework are implemented that assess the extent to which the SS/HS elements, guiding principles, and strategic approaches of the SS/HS Framework are implemented.
Our approach entails examining the impact of collaboration and partnership within these nested sites through a hierarchical approach that describes the implementation within each state program from the state to the LEA and down to the community, as well as conducting a cross-state comparison. Exhibit 4 outlines the variables that will be created and used in the analyses. We anticipate that the results will allow us to test the difference that factors such as State and LEA support, consumer participation, and contextual factors make to program success.
Multiple regression analysis will be used for continuous outcomes, and logistic regression for categorical outcomes to assess the relative contributions of the various activities to the success of the SS/HS State program. At the community level, we will use multiple regression analyses to assess the association between state and LEA support, consumer participation, contextual factors, and program success. The MSE team will also employ a fixed-effects model with dummy variables for each community (minus one) to ensure that community-level factors are controlled for in the model.
The expiration date will be displayed on all data instruments.
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.
SS/HS
Multi-Site Evaluation Page
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-26 |