The NATIONAL Evaluation OF the Substance Abuse and
Mental Health Services administration’s Youth Programs
Supporting Statement
The Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Substance Abuse Treatment (CSAT) is requesting approval from the Office of Management and Budget (OMB) for data collection activities designed to evaluate three grantee programs that are critical to the Agency’s youth portfolio. The three programs include the 2013 Cooperative Agreements for State Adolescent and Transitional Aged Youth Treatment Enhancement and Dissemination (SYT-ED), the 2015 and 2016 Cooperative Agreements for State Adolescent and Transitional Aged Youth Treatment Enhancement and Dissemination Implementation (SYT-I), and the 2015 Cooperative Agreements for State Adolescent and Transitional Aged Youth Treatment Enhancement and Dissemination Planning (SYT-P).
This data collection includes the following tools:
Implementation Interview Guide
Sustainability Interview Guide
Stakeholder Interview Guide
Provider Survey
Focus Group Guide
The, SYT-ED, SYT-I, and SYT-P programs are authorized under Section 509 of the Public Health Service Act, as amended. The programs support SAMHSA’s Trauma and Justice, Recovery Support, and Health Reform Strategic Initiatives. The information provided by this data collection effort and the evaluation as a whole will provide SAMHSA with the necessary information to continue to improve the SYT programs ability to address the three strategic initiatives listed above.
Preventing and treating substance use and mental health disorders are essential to SAMHSA’s mission to reduce the impact of behavioral health conditions in America’s communities. The specific populations (i.e., adolescents, transitional aged youth) targeted by the youth programs face particular behavioral health challenges. In 2013, an estimated 12 percent of adolescents aged 12 to 17 reported alcohol use in the past month, and 6 percent reported binge drinking. The 2013 NSDUH also reported that 43 percent of young adults aged 21 to 25 reported binge drinking in the past month, and 13 percent were heavy drinkers (SAMHSA, 2014a). Approximately 22 percent of young adults aged 18 to 25 and 9 percent of adolescents aged 12 to 17 reported being current illicit drug users. In both age groups, marijuana was the most common drug reported at 19 percent and 7 percent, respectively (SAMHSA, 2014a). In addition, an emerging threat to public health, particularly among America’s youth (aged 12 to 25), is prescription drug misuse, which includes consumption for euphoria (to get high) and/or to self-medicate symptoms of mental and physical illness. Just under 3 percent of individuals aged 12 to 17 have used prescription drugs in the past month that were not prescribed to them, and this rate is nearly double (5.0 percent) among 18- to 25-year-olds. In 2013, approximately 11 percent of youth aged 12 to 17 had a major depressive episode during the past year, and 8 percent had a major depressive episode with severe impairment (SAMHSA, 2014a). Co-occurrence of mental health and substance use disorders also challenges this population. In 2013, approximately one percent of adolescents aged 12 to 17 (359,000 adolescents) had both a substance use disorder and a major depressive episode in the past year (SAMHSA, 2014b). In addition, in 2013 among young adults aged 18 to 25 with any mental illness in the past year, approximately 30 percent also met the criteria for past-year substance use disorder. This percentage was even higher among young adults with serious mental illness with almost 40 percent meeting the criteria for past–year substance use disorder (CBHSQ, 2015).
Substance use and mental health disorders have a substantial impact on the health and well-being of individuals, their families, and their communities. For example, studies show strong links between substance use and decreased ability to learn, deficits in academic performance, increases in school truancy and dropping out, involvement in delinquency and perpetration of aggression, depressive and anxiety disorders, and suicide ideation or attempts (e.g., Cox et al., 2007; Bryant et al., 2003; Brook et al., 2002; Mason and Windle, 2002; DuRant et al., 1999). In addition, disruptive and aggressive behavior problems in youth are associated with a host of developmental problems for the perpetrator and victims of such violence (Arseneault et al., 2006).
Each of the SYT programs helps provide targeted services and evidence-based practices (EBPs) to address the specific needs of adolescents, youth and their families/caregivers. The SYT programs are briefly described below:
2013 SYT-ED grant program focuses on development of a comprehensive service system that supports treatment services, infrastructure, and delivery of EBPs (assessments and treatments) at the provider level (two providers per grantee). Under the SYT-ED initiative, states, territories, and tribes target services for adolescents (aged 12 to 18) and their families or primary caregivers, for transitional aged youth (aged 18 to 24) and their families or primary caregivers, or both.
2015 and 2016 SYT-I grant programs expands on the earlier SAT-ED and SYT-ED grant cohorts by continuing to fund a combination of infrastructure improvement and direct treatment service delivery. These grants are designed to bring together stakeholders across the systems serving the population of focus to strengthen an existing coordinated network. The SYT-I projects are to serve as an example to be replicated—projects are expected to implement successful EBPs (assessments and treatments) at four pilot provider sites that address the needs of the population of focus. The population of focus includes adolescents and transitional aged youth; grantee projects also may choose to provide supportive services to their families or primary caregivers.
2015 SYT-P grant program focuses on the development of a comprehensive strategic plan by states/territories/tribes to improve treatment for adolescents with substance use disorders and/or mental health disorders. The main goal of the plan is to strengthen the existing infrastructure system to help assure that youth have access to evidence-based assessments and treatment models as well as recovery support services. SYT-P is an infrastructure grant and does not include funds for the provision of direct treatment services.
The evaluation contractor has been tasked with conducting a cross-program, multi-site evaluation of SAMHSA’s Youth substance abuse treatment programs over a 5-year period. This project will provide a comprehensive look at the SAT-ED, SYT-ED, SYT-I, and SYT-P programs by conducting three interrelated evaluation efforts: process evaluation, an outcome evaluation, and an economic evaluation. The 2012 SAT-ED cohort is included in the overall evaluation but will not participate in the Implementation, Sustainability, and Stakeholder Interview Guides, Provider Survey, and Focus Group Guide described in this request.
The Implementation, Sustainability, and Stakeholder Interview Guides, Provider Survey, and Focus Group Guide are key components in the cross-site evaluation design for the SYT programs. The evaluation design consists of three interrelated evaluation efforts: a process evaluation, an outcome evaluation, and an economic evaluation. Each of these three evaluation types occurs at multiple levels—grantee, provider, and client—and supports the evaluation/analytic questions (see Attachment 1 for relevant evaluation questions) and tasks that have been identified as priorities for these programs.
Data collection efforts will directly provide qualitative and quantitative data for the process evaluation. The process evaluation component serves the critical role of establishing the overall evaluation’s context and consequently aids in the interpretation of its findings. It focuses on describing the content of the grantees’ programs and their underlying logic models. This contextual information then informs both the outcome and economic evaluations.
The outcome evaluation provides information on the impact that the programs had toward achieving their objectives and on client and family treatment outcomes. The economic evaluation provides information on the resource needs and associated costs of implementing the programs and information on how the programs (or components of the programs) may be sustained after SAMHSA funding has ended. Combined, these three evaluations will provide a comprehensive picture of how the SYT programs were implemented; their effect on grantees/treatment systems, providers, and clients; and what value the programs provided. Additional detail on the overall evaluation methodology, design and analysis plans for the evaluation, telephone interviews and provider survey are included in Section A16.
The SYT-ED, SYT-I, and SYT-P Implementation, Sustainability, and Stakeholder Interview Guides are semi-structured interviews designed to systematically collect data on information related to program implementation facilitators and barriers, infrastructure development, factors related to sustainability, and performance that will inform ongoing recommendations to improve program performance and administration. These Implementation, Sustainability, and Stakeholder Interview Guides were informed by interview guides that were used successfully in other evaluations including the SAMHSA Access to Recovery Evaluation, the Office of the Assistant Secretary for Planning and Evaluation (ASPE) Medicaid Expansion Evaluation, and the SAMHSA Homeless Programs Evaluations.
Interviews will be conducted in person on site visits for five SYT grantees while all other grantees will complete interviews via telephone on teleconferences hosted by the evaluation team. These interviews, both in person and by telephone, are estimated to take approximately 1 hour each.
Grantees and their pilot site providers will be asked to participate in an interview annually while their program is active. They will be asked to complete the implementation interview annually until the last year of the grant program when they will be asked to complete the sustainability interview. The grantees selected for site visits will only be visited once, during which the initial Implementation and Stakeholder Interview Guides will be completed. The Sustainability Interview Guide will be administered by telephone in the last year of their grant program.
All grantees and provider sites involved in the SYT programs are eligible to participate in the implementation interviews. These interviews collect detailed information on the manner in which the programs were implemented and how successfully they were implemented. The Implementation Interview Guide focuses on current treatment service delivery systems, and any needs or gaps in this system, alignment of services with client needs, and facilitators and challenges related to implementation of the SYT programs. SYT-ED and SYT-P grantees and SYT-ED providers will be asked to participate in the implementation interview once during their grant program. SYT-I grantees and providers will be asked to participate twice (once per year) during the course of their grant program.
The Implementation Interview Guide is composed of the following sections: Involvement/Overview, Treatment and Service Delivery System, SYT Program Implementation, Infrastructure and Development, Funding, Facilitators and Challenges, Impact and Utility, and Sustainability. Exhibits 1 and 2 provide brief descriptions of each section and the Implementation Interview Guide can be found in Attachment 2.
Exhibit 1. Grantee Implementation Interview Guide
Involvement/Overview Asks for general information on the grantees’ roles in their state agency and their involvement with the SYT program. |
Treatment and Service Delivery System Asks the grantees about their state/territory/tribes’ treatment and service delivery systems in terms of their abilities to support family-informed care for adolescents and/or transitional youth, the gaps in the current system, and ways that the current SYT program is addressing these needs or gaps. |
Program Implementation Aims to establish a baseline understanding of the SYT program that the grantee has implemented and efforts made towards disseminating EBPs for substance use assessment and treatment for adolescents and/or transitional youth. |
SYT Infrastructure and Development Asks the grantee about the state/territory/tribe-level infrastructure in place prior to receiving the SYT grant and infrastructure-development activities undertaken during the grant. |
Funding Asks the grantee to identify any funds being used to support family-informed treatment for adolescents and/or transitional aged youth apart from the SYT grant funds. |
Facilitators and Challenges Asks the grantee about facilitators and challenges in implementing the SYT program. |
Impact and Utility Identifies any needs or gaps in the existing treatment service system and how the SYT program met these needs or gaps. |
End of Interview Ask grantees to share any additional issues or data that should be reported/collected. |
Exhibit 2. Provider Implementation Interview Guide
Overview Asks for general information on the provider’s organization and its involvement with the SYT program. |
Program Implementation Asks the providers to identify services they provide and who their target population includes and to explain how they implemented their SYT program at their site. Providers are asked to describe any involvement of family members with their clients’ treatment process, client flow, and service delivery process and how the clients can access their organization. |
SYT Infrastructure and Development Asks the providers to explain any start-up or ongoing implementation infrastructure development activities they have engaged in. |
Facilitators and Challenges Asks the grantees about facilitators and challenges in implementing the SYT program and solutions they have applied to overcome these challenges. |
Outcomes Asks the grantees their opinions on how successfully the SYT program has been implemented and if the existing treatment system prior to the program changed since the implementation of SYT. |
(continued) |
Exhibit 2. Provider Implementation Interview Guide |
End of Interview Asks providers to share any additional issues or data that should be reported/collected. |
The purpose of the Sustainability Interview Guides is to understand efforts the grantee and providers have made towards establishing a sustainability plan; what aspects of the SYT programs they found most useful and if they plan on sustaining these aspects; facilitators and challenges to successfully sustaining these programs; and possible key stakeholders involved in program sustainability. Each SYT grantee and provider will be asked to complete the sustainability interview once during the final year of their grant program. Exhibit 3 provides a brief description of each section and the Sustainability Interview Guides can be found in Attachment 3.
Exhibit 3. Grantee and Provider Sustainability Interview Guides
Involvement/Overview Asks for general information on the grantees’ roles in their state/territory/tribe agencies and their involvement with the SYT program. |
Service Delivery (Providers only) Ask for general information about the types of services provided to youth and their families and their sustainability after grant funding. |
Sustainability Establishes a baseline understanding of how grantees/providers plan to sustain their projects after SAMHSA funding ends by identifying sustainability plans, resources needed, factors contributing to or impeding successful sustainability of the program, activities, and the partners involved in sustainability efforts. |
End of Interview Asks grantees/providers to share any additional issues to discuss. |
The purpose of the Stakeholder Interview Guide is to learn about the SYT programs from the perspective of key stakeholders and local funders. The Interview Guide aims to learn about other state/territory/tribe agencies and community-based providers involved in the programs; the ways in which they are involved; and their perspectives on how the programs have been implemented, their impact on and contribution to the community, and efforts made toward sustainability. The key stakeholders will be identified through the SYT grantee and provider implementation and sustainability interviews, as well as through document review and an environmental scan. Stakeholders will be asked to complete the stakeholder interview once during the course of the grant program. Exhibit 4 provides a brief description of each section and the Stakeholder Interview Guide can be found in Attachment 4.
Exhibit 4. Stakeholder Interview Guide
Background/Involvement Asks for general information on the stakeholders’ personal involvement and their organizations’ involvement in the SYT programs. |
Implementation Asks the stakeholders to identify factors that supported or challenged the successful implementation of the SYT programs. |
Impact and Utility Asks the stakeholders how well the SYT program was integrated into the existing substance abuse treatment system for youth and their families; how successfully it has been implemented; and its impact on health disparities, access to and use of EBPs, client-level outcomes, and family or caregiver involvement. |
Facilitators, Challenges, Outcomes and Lessons Learned Asks the stakeholders to identify facilitators and challenges regarding program infrastructure development at the state-, territory-, or tribe-level and implementation or service delivery at the provider sites. Also asks for input on outcomes and impact added by the SYT grant at the stakeholder’s organization and the state, territory, or tribe. |
Sustainability Asks the stakeholders about their involvement in any SYT sustainability planning and what they think the effect would be if the program were not sustained. |
End of Interview Asks stakeholders to share any additional issues to discuss. |
Substance abuse service provider organizations (e.g., treatment facilities implementing EBPs) participating as pilot sites in the SYT-ED or SYT-I grant are eligible for the SYT Provider Survey. The Provider Survey aims to collect data to help identify program activities and services that are being implemented as part of the SYT-ED/SYT-I grant programs and the effect these activities or services may have on client outcomes and treatment systems.
The Provider Survey will collect data on linkages with the grantee and within the system for providing services and a safety net to adolescents, transition age youth, and their families. Topics around grantee dissemination and outreach efforts as well as EBP, other training activities and the economic cost of services will also be explored.
The Provider Survey was developed to specifically address the unique provider organization experiences associated with participation in a SYT-ED and SYT-I grant. The survey format and questions are based on a successful provider survey used for the SAMHSA Access to Recovery evaluation effort (OMB 0930-0299). The survey questions included in the Provider Survey link to the evaluation’s questions and are developed by experts on the SYT cross-site evaluation team.
Providers will be asked to complete the Provider Survey once per year during the course of the grant program. Exhibit 5 provides a brief description of each section and the Provider Survey can be found in Attachment 5.
Exhibit 5. Provider Survey
Organizational Background Includes questions on organizational background. |
Client Population and Services Provided Gathers aggregated client information including the types of clients served and the services offered. |
Project/Program Implementation Activities Includes questions on the activities providers engaged in to implement the providers’ SYT programs. |
Sources of Funding and Outside Partnerships Asks providers about the different sources of funding used to provide services and the partnerships with other stakeholders in their community. |
Resources Used and Costs |
Asks providers about the labor, building and facilities, contract services, supplies, materials and any other resource used to implement their SYT program. |
Wrap Up Asks the provider to rate their satisfaction with the SYT program and to provide any suggestions. |
Exhibit 6. Focus Group Guide
Impact Includes a question on the impact of implementing EBPs in the provider’s community. |
Implementation Effectiveness Asks clinicians/staff members about the effectiveness of EBPs on clients, families and staff. |
Leadership Engagement Includes questions around the commitment of the provider organization’s leadership to the successful implementation of EBPs. |
Implementation Climate Targets the expectations of provider organization leadership, and asks whether those expectations were met. |
Provider Collaborative |
Asks about the role of other community providers in the implementation of EBPs by the grantee provider organization. |
The grantee, provider, and stakeholder interviews are designed as both in-person and telephone-based interviews and the same procedures will be followed for both interview methods. In both settings, interviewers will ask respondents questions, and note takers will record responses into a word document. Five grantees will complete the initial interviews (both grantee and provider) in person while the remaining interviews will be conducted via telephone to reduce overall costs. Qualitative interviews are still necessary to capture the often nuanced and sometimes complex nature of program implementation and sustainability. The completed interviews will be reviewed by the interviewer for accuracy. Interviews will also be audio recorded with the respondent’s permission. All recordings will be stored on RTI International’s secure servers and destroyed once de-identified transcriptions are completed.
The Provider Survey will primarily be administered through a web instrument to an identified key informant at the provider organization (e.g., program director, clinical supervisor). Using a web instrument allows for automated skip procedures and automated fill-ins based on prior responses to certain questions, which will significantly reduce the burden among subsets of respondents. This method also uses automated data entry and greatly reduces the possibility of data entry error. Key informants who are unable to complete the web instrument will have an option to receive a paper version through the mail; however, experience with web surveys suggest that respondents will find the web instrument to be the less-burdensome method of completing the survey. The Provider Survey web instrument will comply with the requirements of Section 508 of the Rehabilitation Act.
Focus Group Guide
Like the interviews, focus groups will be conducted in-person with providers supporting grantees from five visited sites, while the remaining focus groups will be done virtually with providers supporting non-visited grantees. Virtual focus groups will be conducted with teleconferencing and webinar technology hosted by the evaluation team. This approach will balance the need to collect detailed quantitative data on implementation with the need to limit burden and cost to the evaluation team as a whole. In both settings, focus group leaders will ask respondents questions, and a note taker will record responses into a word document. The completed focus group notes will be reviewed by the focus group leader for accuracy. Focus groups also will be audio recorded with the respondents’ permission. All recordings will be stored on RTI International’s secure servers and destroyed once de-identified transcriptions are completed.
Data will be stored electronically and will be accessible by evaluation team members who will be made aware of what data are available. This reduces the possibility of any unnecessary contact with participants to collect redundant information.
Data collected through the Implementation, Sustainability, and Stakeholder Interview Guides, Provider Survey, and Focus Group Guide are specific and unique to each grantee program. As such, the data collection tools do not duplicate any other data collection effort. Data collected in this evaluation are not available from other sources and will be unique because of the scale and breadth of the initiative’s implementation: nationwide, across a spectrum of programs, and across a broad cross-section of populations.
Currently, SAMHSA monitors client-level outcomes Government Performance and Results Act (GPRA) (OMB No. 0930-0208) for all grantees and monitors SYT grantees through bi-annual reports (OMB 0930-0344). These data will be sufficient to monitor grantee programs but not sufficient for understanding the context in which each grantee operates.
The SYT bi-annual report template was reviewed by the evaluation team, integrated into the data collection plan and the Implementation Interview Guide was designed to enhance and expand upon the systematic program data collected by the bi-annual reports. For example, the SYT Infrastructure and Development section in the Implementation Interview Guide asks open ended questions to better understand the contextual issues around the infrastructure and development activities identified in the bi-annual reports. With the interview data, the bi-annual report data will be more useful to the evaluation and SAMHSA. Moreover, the Sustainability and Stakeholder Interview Guides, Provider Survey, and Focus Group Guide collect data not covered in the bi-annual reports and include different respondents (e.g., stakeholders, providers).
There will not be a significant impact on small entities. Grantees in the SAMHSA SYT programs include state agencies, local services providers, and tribal organizations and some may be small entities; however, the interviews, provider survey and focus groups are designed to include only the most pertinent information needed to be able to effectively carry out the evaluation.
The Implementation, Sustainability, and Stakeholder Interview Guides will be conducted with grantee project directors, key management staff, provider staff, and key stakeholders once per year to more fully understand how grantee programs develop and any ongoing challenges. The data collection will be aligned with the grant program length such that each cohort will complete annual interviews starting with the implementation interview and finishing with the sustainability interview in the final year of the each grant program. SYT-ED and SYT-P grantees and SYT-ED providers will complete the implementation interview and sustainability once during the course of the grant, while SYT-I grantees and providers will complete the implementation interview twice (once per year) and the sustainability interview once. The five sites selected for site visits will only be visited once and any subsequent interviews will be conducted via telephone.
The Provider Survey will be conducted once with SYT-ED providers and once every year with SYT-I providers. The information collected by the Implementation, Sustainability, and Stakeholder Interview Guides and survey is necessary for a comprehensive cross-site evaluation; without these data, SAMHSA and the evaluation team would not be able to assess specific outcomes of the SYT-ED and SYT-I programs. For the SYT-I providers, repeating data collection annually will provide a unique look at how programs develop and evolve. The Provider Survey will account for the repeated data collection efforts, and select questions where no change is expected will not be repeated to help ensure only relevant information is collected.
The focus groups will be conducted once with each SYT-ED and SYT-I pilot site provider during the evaluation study. Data collected in the focus groups will provide the clinicians’/staff members’ perspectives on the program which is not directly captured by the Implementation, Sustainability, and Stakeholder Interview Guides or the Provider Survey. These data will help answer evaluation questions around implementing EBPs. Clinicians/staff members are uniquely qualified to answer implementation questions on a client, staff and community level. The collected information will provide context to data aggregated from the Implementation, Sustainability, and Stakeholder Interview Guides and Provider Survey.
This information collection fully complies with the guidelines in 5 CFR 1320.5(d) (2).
The notice required by 5 CFR 1320.8(d) was published in the Federal Register on April 20, 2016 (81 FR 23320). No comments were received.
Subject-matter experts on the RTI evaluation team have provided feedback on all aspects of the evaluation and their comments and suggestions were incorporated into the development of the Implementation, Sustainability, and Stakeholder Interview Guides, Provider Survey, and Focus Group Guide. The list of experts is provided in Exhibit 7.
Exhibit 7. RTI Team Experts
Expert |
Affiliation |
Contact Information |
Arnie Aldridge, PhD Research Economist/Quantitative Researcher
|
RTI International |
3040 East Cornwallis Road Post Office Box 12194 Research Triangle Park, NC 27709-2194 |
Olivia Silber Ashley, DrPH Director of Risk Behavior and Family Research Program
|
RTI International |
3040 East Cornwallis Road Post Office Box 12194 Research Triangle Park, NC 27709-2194 |
Laura J. Dunlap, PhD Director of Behavioral Health Economics Program |
RTI International |
3040 East Cornwallis Road Post Office Box 12194 Research Triangle Park, NC 27709-2194 |
|
|
(continued)
|
Exhibit 7. RTI Team Experts |
|
|
Bryan Garner, PhD Senior Implementation Research Scientist |
RTI International |
3040 East Cornwallis Road Post Office Box 12194 Research Triangle Park, NC 27709-2194
|
Alison Green* Director of Adolescent Research and Services |
University of Arizona-Southwest Institute for Research on Women |
181 S. Tucson Blvd., Ste. 101 Tucson, AZ 85716 |
Saira Haque, PhD Senior Health Informaticist/Qualitative Researcher |
RTI International |
3040 East Cornwallis Road Post Office Box 12194 Research Triangle Park, NC 27709-2194 |
Danielle Laborde, PhD President and Senior Epidemiologist |
Hermes, LLC |
6576 Towles Road Wilmington, North Carolina 28409 |
Sally Stevens, PhD Executive Director; Southwest Institute for Research on Women |
University of Arizona-Southwest Institute for Research on Women |
925 N. Tyndall Tucson, AZ 85721-0438 |
* Local evaluator for current SYT-ED grantee
No cash incentives or gifts will be given to respondents.
Concern for privacy and protection of respondents’ rights will play a central part in the implementation of all study components. The evaluation team led by RTI has extensive experience protecting and maintaining data from collection through analysis and reporting.
The evaluation team will safeguard project directories and analysis files containing completed interview and survey data to ensure that there is no inadvertent disclosure of study data. In addition, all studies involving human subjects are reviewed and approved by RTI’s Institutional Review Board (IRB) (Federal Wide Assurance Number 3331). In keeping with 45 CFR 46, Protection of Human Subjects, the procedures for data collection, consent, and data maintenance are formulated to protect respondents’ rights. Strict procedures will be followed for protecting the privacy of respondents’ information and for obtaining their informed consent. Any necessary changes to the cross-site evaluation protocol will be reviewed by the RTI IRB.
The information reported by respondents through the Implementation, Sustainability, and Stakeholder Interview Guides, Provider Survey, and Focus Group Guide is not sensitive personal information as these instruments focus only on programmatic details of the grantee’s program or the providers’ experience implementing these programs. At the start of each interview and focus group and at the beginning of the web survey, respondents will be provided full information on how their response will be used for the evaluation and told that their participation is voluntary. Further, respondents will be informed how their responses will be kept private.
The Implementation, Sustainability, and Stakeholder Interview Guides, Provider Survey, and Focus Group Guide have been deemed non-human subjects research by the RTI’s IRB (FWA #3331) as the data collected is de-identified and does not ask questions about the respondent’s history or behaviors.
Exhibit 8 shows the estimated annualized total hour burden for the Implementation, Sustainability, and Stakeholder Interview Guides, Provider Survey, and Focus Group Guide summed across the SYT-ED, SYT-I, and SYT-P grantee cohorts. The Provider Survey is estimated to take 1 hour for respondents to complete including time to review the survey’s introduction and information on respondent rights and all directions. The Implementation, Sustainability, and Stakeholder Interview Guides are expected to take no more than 1 hour and grantee, provider, and stakeholder organization representatives will participate in the interviews. As described above, five sites will complete some Implementation, Sustainability, and Stakeholder Interview Guides and Focus Group Guide in person while the remaining interviews and focus groups will be done over the phone. Burden is expected to be the same for interviews and focus groups conducted in person or done via telephone. Estimated annualized total burden is estimated at 767 hours with an estimated annual total cost of $35,136.27. Burden hours are also calculated for each instrument below in Exhibit 9.
Exhibit 8. Estimated Annualized Total Burden by Cohort
Grantee Cohort |
Number of Respondents |
Responses per Respondent |
Total Number of Responses |
Hours per Responsea |
Total Burden Hours |
Average Hourly Wageb |
Total Respondent Costc |
SYT-ED grantees |
286 |
1 |
286 |
1 |
286 |
$45.81 |
$13,101.66 |
SYT-I grantees |
377 |
1 |
377 |
1 |
377 |
$17,270.37 |
|
SYT-P grantees |
104 |
1 |
104 |
1 |
104 |
$4,764.24 |
|
TOTAL |
767 |
-- |
767 |
-- |
767 |
-- |
$35,136.27 |
a Hours per response is an average annualized estimate.
b Average hourly wage is calculated using wages from the Bureau of Labor Statics and taking a weighted average of the national average wage for Social and Community Manager and Medical and Health Services Manager.
c Total respondent cost is calculated as hourly wage × hours per response × total number of responses.
Exhibit 9. Estimated Annualized Total Burden by Data Collection Instrument/Activity
Instrument/ Activity |
Number of Respondents |
Responses per Respondent |
Total Number of Responses |
Hours per Responsea |
Total Burden Hours |
Average Hourly Wageb |
Total Respondent Costc |
Sustainability Interviews |
98 |
1 |
98 |
1 |
98 |
$45.81 |
$4,489.38 |
Implementation Interviews |
124 |
1 |
124 |
1 |
124 |
$5,680.44 |
|
Stakeholder Interviews |
183 |
1 |
183 |
1 |
183 |
$8,383.23 |
|
Provider Survey |
74 |
1 |
74 |
1 |
74 |
$3,389.94 |
|
Focus groups |
288 |
1 |
288 |
1 |
288 |
$13,193.28 |
|
TOTAL |
767 |
-- |
767 |
-- |
767 |
-- |
$35,136.27 |
a Hours per response is an average annualized estimate.
b Average hourly wage is calculated using wages from the Bureau of Labor Statics and taking a weighted average of the national average wage for Social and Community Manager and Medical and Health Services Manager.
c Total respondent cost is calculated as hourly wage × hours per response × total number of responses.
There are no respondent costs for capital or start-up or for operation or maintenance.
The total estimated cost to the government for the data collection is $46,225.92. This includes approximately costs for programming, contractor labor for conducting interviews, focus groups, and the Provider Survey and analyzing and reporting data, travel for site visits, and housing and maintaining data.
Approximately $2,468 per year represents SAMHSA costs to manage and administer the survey for 2% of one employee (GS-13, $123,406)
This is a new data collection.
Exhibit 10 outlines the key time points for the study and for the collection of information. The requested period also allows for training and start-up activities associated with the preparation for data collection. All data collection activities will be contingent on OMB approval.
Exhibit 10. Time Schedule for Data Collection
Activity |
Time Schedule |
Receive OMB approval |
Summer 2016 |
Conduct Interviews and Focus Groups SYT-ED providers SYT-I providers SYT-P Providers |
2016 through late 2017 2016 through mid-2019 2016 through mid-2018 |
Conduct Provider Survey SYT-ED providers SYT-I providers SYT-P Providers |
2016 through late 2017 2016 through mid-2019 2016 through mid-2018 |
Prepare Interview, Provider Survey and Focus Group datasets and conduct initial analysis |
Late 2017 through mid-2019 |
Finalize analysis and prepare preliminary reports |
Mid-2018 through 2019 |
Prepare final evaluation reports and distribute key findings |
2019 |
The evaluation of the SYT-ED, SYT-I, and SYT-P programs will use quantitative and qualitative data from multiple sources. The evaluation approach leverages existing quantitative data collected as a result of local evaluations and assessments conducted by the grantees including GPRA, GAIN, and other assessment tools that track the performance of their program and program clients. The evaluation also will use additional quantitative data concerning program performance (e.g., type and amount of services provided via web survey for the SYT‑ED and SYT-I grantees) collected for the purpose of this evaluation. In addition, existing (e.g., grantee reports to SAMHSA) and newly collected qualitative data—primarily the Implementation, Sustainability, and Stakeholder Interview Guides, Provider Survey, and Focus Group Guide —will be used. These data will assess the process by which the SYT programs are implemented and the outcomes of these programs. Leveraging existing data and supplementing data gaps with primary data collection allow us to obtain comprehensive data to accomplish the goals and objectives of the proposed evaluation of the SYT programs.
The fundamental study design is pre-post, comparing baseline outcomes with follow-up outcomes (typically 6 months later). The presumption is that program treatment and services will cause outcomes to improve by the time they are measured at later interviews. When data support additional longitudinal models (e.g., multiple GAIN assessments after baseline, 3-month GPRA among adolescent and transitional aged youth programs), the post-baseline interviews will be treated as program outcomes and will be used to estimate changes in outcome trajectories. The rigor of the pre-post design depends on the validity of the assumption that interim treatment explains all changes in client outcomes from baseline to follow-up. This is a strong assumption that ignores many confounders and processes (such as regression to the mean) that are viable alternative rationales for any changes over time implying that pre-post estimates will be biased. For this reason, we also plan to identify quasi-experimental comparison groups to reduce bias from these sources. Quasi-experimental comparison groups will be analyzed with propensity score methods using an intent-to-treat assumption.
Many of the components we will examine within statistical models are based on qualitative data, particularly from data gathered by the Implementation, Sustainability, and Stakeholder Interview Guides, Provider Survey, and Focus Group Guide. For example, consistent challenges to implementation may emerge from key informant interviews that we will wish to analyze within a statistical model of client outcomes (moderation). Example table shells for Provider Survey data which present descriptive characteristics which may be incorporated into statistical models can be found in Attachment 9. For the interviews and focus groups, formal qualitative analyses will also aid in producing taxonomies of program models; lessons learned; project narratives; challenges and facilitators, and innovations, EBPs, and adaptations. Our fundamental approach will be inductive. However, because the projects may exhibit a complex set of similarities and differences, we propose the use of Boolean-based Comparative Qualitative Analysis (CQA) to rigorously distill complex data. CQA supports the development of hypotheses around the causal chain of project implementation through long-term client outcomes.
OMB approval expiration dates will be displayed.
There are no exceptions to the certification statement. The certifications are included in this submission.
Arseneault L, Walsh E, Trzesniewski K, Newcombe R, et al. (2006). Bullying victimization uniquely contributes to adjustment problems in young children: A nationally representative cohort study. Pediatrics, 118(1), 130-138.
Brook, DW, Brook, JS, Zhang, C, Cohen, P, & Whiteman, M. (2002). Drug use and the risk of major depressive disorder, alcohol dependence, and substance use disorders. Archives of General Psychiatry, 59(11), 1039-1044.
Bryant, AL, Schulenberg, JE, O’Malley, PM, Bachman, JG, & Johnston, LD. (2003). How academic achievement, attitudes, and behaviors relate to the course of substance use during adolescence: a 6-year, multiwave national longitudinal study. Journal of Research on Adolescence, 13(3), 361-397.
Center for Behavioral Health Statistics and Quality. (2015). Behavioral health trends in the United States: Results from the 2014 National Survey on Drug Use and Health (HHS Publication No. SMA 15-4927, NSDUH Series H-50). Retrieved 02/08/2016 from http://www.samhsa.gov/data/
Cox, RG, Zhang, L, Johnston, WD, & Bender, DR. (2007). Academic performance and substance use: findings from a state survey of public high school students. Journal of School Health, 77(3), 109-115.
DuRant, RH, Smith, JA, Kreiter, SR, & Krowchuk, DP. (1999). The relationship between early age of onset of initial substance use and engaging in multiple health risk behaviors among young adolescents. Archives of Pediatric Adolescent Medicine, 153(3), 286-291.
Mason, AW & Windle, M. (2002) Reciprocal relations between adolescent substance use and delinquency: a longitudinal latent variable analysis. Journal of Abnormal Psychology, 111(1), 63-76.
Substance Abuse and Mental Health Services Administration, Center for Behavioral Health Statistics and Quality. (June 10, 2014a). The CBHSQ Report: A Day in the Life of Young Adults: Substance Use Facts Update. Rockville, MD.
Substance Abuse and Mental Health Services Administration, Center for Behavioral Health Statistics and Quality. (September 4, 2014b). The NSDUH Report: Substance Use and Mental Health Estimates from the 2013 National Survey on Drug Use and Health: Overview of Findings. Rockville, MD.
ATTACHMENTS
Attachment 1 – Evaluation Questions
Attachment 2 – Implementation Interview Guides
Attachment 3 – Sustainability Interview Guides
Attachment 4 – Stakeholder Interview Guides
Attachment 5 – Provider Survey
Attachment 6 – Focus Group Guide
Attachment 7 – Interview Guide Supporting documents
Attachment 8 – Provider Survey Supporting documents
Attachment 9 – Provider Survey table shells
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Orme, Stephen |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |