Supporting Statement
Part A. Justification
The Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Substance Abuse Prevention (CSAP) requests OMB approval to collect a new community outcomes data for the cross-site evaluation of the Strategic Prevention Framework State Incentive Grant (SPF SIG) program, Cohorts IV and V. The community outcome data will inform the broader cross-site evaluation. The instruments seeking approval are the: 1) SPF SIG National Outcome Measures and 2) Community Outcomes Data Fields.
CSAP has funded two cross-site evaluations of the Strategic Prevention Framework State Incentive Grant (SPF SIG), one focused on Cohorts I and II and the other on Cohorts III, IV, and V. Collectively, these evaluations provide an important opportunity to inform the prevention field on current practices and their association with community- and state-level outcomes.
The primary cross-site evaluation objective is to determine the impact of SPF SIG on building prevention capacity and infrastructure, and preventing the onset and reducing the progression of substance abuse, as measured by the SAMHSA National Outcomes Measures (NOMs). Data are collected at the grantee, community, and participant levels. The collection of community
outcomes data is the focus of the current request.
The SPF SIG is a major SAMHSA grant program that supports an array of activities to help grantees and subrecipient communities build a solid foundation for delivering and sustaining substance abuse prevention services that are effective in reducing the incidence and prevalence of substance use as well as improving risk and protective factors associated with substance use. CSAP provides funding to states/territories, jurisdictions, and tribal entities to implement the five steps of the strategic prevention framework (SPF), which are:
Step 1: Profile population needs, resources, and readiness to address the problems and gaps in service delivery;
Step 2: Mobilize and/or build capacity to address needs;
Step 3: Develop a comprehensive strategic plan;
Step 4: Implement evidence-based prevention programs, policies, practices and infrastructure development activities; and
Step 5: Monitor process, evaluate effectiveness, sustain effective programs/activities, and improve or replace those that fail.
For the purposes of this document, the word grantee will refer to all funded states, jurisdictions, and tribal entities. SPF SIG grantees select and fund their own community subrecipients who actually implement the prevention interventions. The selection of these subrecipients is not guided by CSAP.
Information on both cross-site evaluations is presented below as context for understanding SAMHSA’s request to collect community outcomes data from Cohorts IV and V.
A1b1. Cohorts I, II, and III Cross-site Evaluation
The National Institute on Drug Abuse (NIDA) provided support to CSAP to evaluate the impact of the SPF SIG project for Cohorts I and II. Specifically, data were collected from the 26 states and territories receiving grants initiated in 2004 (Cohort I sites) and 2005 (Cohort II sites) and up to 32 non-Cohort I and II grantee states and territories that served as a comparison group. Data from a Community-Level Instrument was used to determine the impact of the SPF SIG on all of the NOMs domains related to prevention (i.e., Abstinence, Education/Employment, Crime and Criminal Justice, Access/Capacity, Retention, Cost Efficiency and Use of Evidence-based Practices). The evaluation has also measured the effect of establishing and sustaining infrastructure at the state and community-levels to allow for data-based decision-making; the implementation of the SPF; and the environmental factors that affect substance abuse.
A1b2. Cohorts IV, and V Cross-site Evaluation
The contractor is implementing a multi-level evaluation design encompassing process and outcome data collection at the grantee, community, and participant levels. Data have been gathered from the 16 states, jurisdictions, and tribal entities receiving grants in FY 2006 (Cohort III), the 25 Cohort IV grantees funded in FY 2009 (one subsequently relinquished funding), and the 10 Cohort V grantees funded in FY 2010. Data collection for Cohort III has ended; therefore, this data collection request pertains only to Cohorts IV and V. Funding ended for most Cohort IV grantees in June 2015.
Collection of community data was a part of grantees’ strategic plans and most collected (or, for Cohort V, are continuing to collect) these data for their grantee-level evaluations. Grantees gathered existing archival or survey data (e.g., student risk behavior survey, required Federal reporting of arrests and motor vehicle crashes) for their data where available. Some tribal and jurisdiction grantees lacked such data and conducted primary data collection for their SPF SIG evaluations.
The purpose of the SPF SIG grant program is to improve substance abuse prevention systems and enhance the quality of prevention programs, primarily through the implementation of the SPF process. The goal of this initiative is to provide states, jurisdictions, tribal entities, and the communities within them with the tools necessary to develop an effective prevention system with attention to the processes, directions, goals, expectations, and accountabilities necessary for functionality. SAMHSA/CSAP needs to collect information over the course of the remaining grant period to monitor the progress of the SPF SIG initiative. CSAP will use the findings from the analysis of the community outcomes data in the cross-site evaluations to assess the impact of SPF activities on community-level outcomes. Without these data, the extended impact of the SPF process across various types of communities would remain unknown. Additionally, findings from this cross site evaluation will assist CSAP policymakers and program developers as they design and implement future initiatives.
Cohorts IV and V grantees will provide aggregate data for at least one substance use consumption, consequence, and/or mediating/intervening variables for each of their subrecipient communities. (In the case of twelve single-community grantees, data are reported for the entire jurisdiction or tribal entity.) Grantees that have data sources available annually will provide approximately six annual data points per measure: a baseline time point prior to subrecipient or intervention funding, and follow-up time points at least 11 months after the previous time point for the duration of funding. At a minimum, grantees with less available data (e.g., tribal grantees that conducted primary data collection for their evaluations) will provide a baseline time point and a follow-up time point close to the end of the grant period.
Reported measures must be a SPF SIG National Outcome Measure (see Attachment A) or a substitute measure. Community outcomes data are to be drawn from existing survey or administrative data within the state, tribe, or jurisdiction that grantees have used for their SPF SIG grantee-level evaluations. In addition to providing the measured value and variability estimates for each outcome measure, the contractor will request measure descriptions, including targeted SPF SIG priority, data source name and type, data collection time frame, item and response option wording, and population and sample information. See Attachment B for a full list of proposed data collection fields.
The contractor team will create an Excel data entry spreadsheet for each grantee. Given the non-confidential nature of aggregate community outcomes data, the data entry spreadsheet will be distributed to grantees and returned to the contractor via email. This spreadsheet will be pre-populated with any community outcomes data that the contractor has in our cross-site database. Each time point for each outcome for each subrecipient community will be entered in its own row in the spreadsheet.
Information technology has been used to minimize respondent burden for Cohorts IV and V in the SPF SIG Cross-site Evaluation. An Excel data entry spreadsheet will be created for each grantee. The Excel format will allow grantees to copy and paste data elements that are applicable across multiple time points and multiple subrecipients. Closed-ended fields will contain drop-down menus that allow for the quick selection from among response options. Given the non-confidential nature of aggregate community outcomes data, the data entry spreadsheet will be distributed to grantees and returned to the contractor via email, which will be both efficient and expedient.
Technology is also used to facilitate communication and provide updates to SPF SIG personnel. Through a SPF SIG listserv, grantee evaluators, project directors, coordinators, and other key staff will have the opportunity to exchange valuable advice; find guidance and resource materials; and receive announcements and clarifications from CSAP, other SPF SIG grantees, and the cross-site evaluation team. In addition to the listserv, grantees can access community outcomes guidance materials on the contractor’s Knowledge Base and the cross-site evaluation team will also send electronic copies of guidance materials via email to SPF SIG grantees upon request.
The information being collected for the cross-site evaluations for SPF SIG Cohorts IV and V is specific to the program and is not available elsewhere.
SPF SIG grantees are mainly State agencies, tribal organizations and other jurisdictions; they are not typically small entities. Some subrecipients may be small entities; however, the impact of reporting pre-existing community outcomes data should not be significant.
The requested Cohort IV and V community outcomes data collection will be a one-time data collection, occurring for approximately two months following the receipt of OMB approval.
If these data are not collected from the grantees, SAMHSA will have an incomplete dataset for the entire program and will lose critical information. This is particularly problematic for tribal grantees (who comprise a significant portion of the Cohort IV and V grantees) and jurisdictional grantees, neither of whom are represented in the national datasets (reported at the state level) that the cross-site evaluation is using to assess outcomes at the grantee level (e.g., National Survey on Drug Use and Health; Fatality Analysis Reporting System).
This information collection fully complies with 5 CFR 1320.5(d) (2).
The notice required in 5 CFR 1320.8(d) was published in the Federal Register on December 4, 2015 (80 FR 75869).
The community outcomes data fields were reviewed by SAMHSA staff and contractors. These experts provided feedback on the data collection instrument and the instrument was revised based on their feedback. The following individuals consulted with CSAP staff on the community outcomes data fields:
Nilufer Isvan, Ph.D.
DACCC Data Analysis Team Lead
Human Services Research Institute
Cambridge, MA
Virginia Mulkern, Ph.D.
DACCC Project Director
Human Services Research Institute
Cambridge, MA
Kelly Vander Ley, Ph.D.
Senior Research Associate
RMC
Portland, OR
Kelly Jarvis, Ph.D.
Senior Research Associate
RMC
Portland, OR
Gillian Leichtling
Senior Research Associate
RMC
Portland, OR
There is no payment to any respondent.
No individual-level or personal data will be collected; grantee staff will provide aggregate community data. Although not collecting individual-level data, evaluation staff are trained on the importance of privacy and in handling sensitive data. SAMHSA and its contractors will not receive identifiable client records. A PIA and SORN application have been submitted for this data collection for review at the U.S. Department of Health and Human Services
Grantees and subrecipients will potentially report data derived from sensitive questions, such as those about substance use behavior. However, these data are generally collected for a purpose other than SPF SIG (e.g., student risk behavior survey, required Federal reporting of arrests and motor vehicle crashes) and will only be reported to the contractor in the aggregate.
All 34 SPF SIG Cohort IV (n=24) and Cohort V (n=10) grantees are expected to provide community data. Grantees will provide one response that includes data for all available time points. The community outcome instrument (completed in the form of a spreadsheet) is estimated to take approximately 4 hours to complete; this includes time to look up and compile information (3 hours) and time to complete the spreadsheet (1 hour). The estimated burden time is based on test instruments completed by evaluation staff members that have experience working with SPF SIG grantees (see Section B.4 for more detail). There are no direct costs to respondents other than their time to complete the instrument. Table 1 provides the details of the total burden for the instrument (submitted only once): burden hours estimated at 136, and respondent cost of $5,559.68 (total burden hours × the average hourly wage for State government managers, as reported in the 2014 Occupational Employment Statistics [OES] by the Bureau of Labor Statistics [BLS]; see http://www.bls.gov/oes/current/naics4_999200.htm#11-0000).
Table 1. Estimates of Annualized Hour and Cost Burden to Respondents for Cohorts IV and V
Respon-dent |
No. of Respon-dents |
No. of Responses per Respondent |
Total No. of Responses |
Burden per Response (Hrs.) |
Total Burden (Hrs.) |
Hourly Wage Cost |
Total Cost |
Grantee |
34 |
1 |
34 |
4 |
136 |
$40.88 |
$5,559.68 |
There are no capital/startup costs or operational/maintenance of services costs associated with this project.
The total estimated cost to the government for this one time data collection is $256,472.68. This includes approximately $89,711 for developing the data collection spreadsheets; collecting data from grantees; processing, cleaning, and housing data; and analyzing and reporting data. Approximately $55,602 per year represents SAMHSA costs to manage/administer the data collection and analysis for 25% each of two employees (GS-14-10, $111,203 annual salary). Approximately $105,600 per year represents SAMHSA costs to monitor and approve grantee reporting (10% time of 10 Project Officers at $105,600 annual salary) and approximately $5,559.68 for grantee data collection.
This is a new collection of information.
Time Schedule
Table 2 outlines the key time points for the SPF SIG community outcomes data collection.
Table 2. Time Schedule for SPF SIG Community Outcomes Data Collection
Activity |
Time Schedule |
Prepare for data collection, including preparation of data collection spreadsheets |
April 2016 |
Obtain OMB approval for data collection |
April 2016 |
Collect data |
May 2016-June 2016 |
Process data |
July 2016-September 2016 |
Analyze data and disseminate findings |
October 2016–September 2018 |
Note. OMB, Office of Management and Budget; SPF SIG, Strategic Prevention Framework State Incentive Grant
Analysis
Community outcomes data will be used to address two evaluation research questions in the SPF SIG cross-site evaluation:
Did the implementation of the SPF SIG lead to community-level improvement on NOMs or other outcomes?
What factors accounted for variation in performance on NOMs and other outcomes across funded communities?1
Evaluation Research Question 1 will be examined using meta-analysis. Meta-analytic techniques will be used to calculate effect sizes that will provide descriptive information regarding the magnitude of change in outcomes. We will calculate effect sizes for community outcomes as follows: For each of the outcome measures at each time point, we will calculate odds to reflect the occurrence of a “successful event.” For instance, if the baseline percentage of arrests due to alcohol/drugs is 25%, then the percentage of arrests that are not due to alcohol/drugs is 75%. Thus, the odds of a successful event (i.e., an arrest not related to alcohol or drugs) are calculated as 3.0 (3 successes to every 1 failure). From these odds, an odds ratio will be computed to reflect the amount of change in outcomes over time, specifically by dividing the odds of a successful event at follow-up by the odds of a successful event at baseline. Odds ratios are centered at 1.0; odds ratios greater than 1.0 indicate greater odds of success at follow-up compared with baseline (i.e., improvement), whereas odds ratios less than 1.0 indicate greater odds of success at baseline (i.e., worsening). All subsequent analyses will be conducted on the natural log of the odds ratio,2 and variance estimates will be calculated.
Evaluation Research Question 2 will be examined via hierarchical linear modeling (HLM) or meta-regression. SPF SIG data represent data in which observations are nested—for instance, communities nested within grantees. To address the nested nature of these data, the contractor plans to use HLM, a multilevel approach that can accommodate the nested design as well as the addition of covariates. The HLM approach develops regression equations at each level of nesting to account for variation at different levels of the model. Each subrecipient will have its own estimated trajectory for the outcome variable, which will vary across subrecipients and grantees. This approach has a number of advantages over traditional general linear models (GLMs), including the ability to include individual- and cluster-level covariates while adjusting for random effects associated with each cluster and greater flexibility regarding covariance structures.3
When HLM is not possible because of small sample sizes or failure to meet statistical assumptions, meta-regression will be the technique of choice. Meta-regression is a technique in which regression analyses are conducted on effect sizes (e.g., odds ratios) rather than on raw data. Random-effects meta-regression models allow for heterogeneity in the “true effect” and thus allow for the likelihood that differences in grantee characteristics may result in differing effect sizes from one grantee to another.4 This method treats each effect size as the result of a different “study” conducted on a grantee or on a community, where each study (i.e., grantee mean or community mean) is weighted by the inverse of its variance. Modified regression analyses are then conducted on these weighted means, in a manner conceptually similar to that of a traditional regression. One limitation to the meta-regression approach (similar to traditional regression) is that it is not recommended when the number of studies (i.e., grantees or subrecipient communities) is small. Just as with traditional regression analyses, a typical rule of thumb is at least 10 studies per covariate, although more are better. As a result, meta-regression models will have to remain relatively simple for cohort-level analyses, particularly at the grantee level, and may be more complex when the sample size is larger, such as for community-level or cross-cohort analyses. If inferential statistics are underpowered, we will either cautiously conduct inferential statistics and focus interpretation on the effect size estimates and confidence intervals rather than p-values or restrict our analyses to descriptive statistics (in the event of very small sample sizes).
Publications
In combination with other cross-site evaluation data sources, the contractor will use SPF SIG community outcomes data to help SAMHSA reach its diverse stakeholders through targeted products and innovative dissemination venues. The objective for all reports and dissemination products is to provide user-friendly documents and presentations that help SAMHSA successfully disseminate and explain the findings. The dissemination plan includes products in a variety of formats for a variety of target audiences. Audiences for these reports will include Congress, SAMHSA Centers, the evaluation’s SAMHSA Contracting Officer’s Representatives (CORs), SPF SIG grantees, and the broader substance abuse prevention field (e.g., academia, researchers, policy-makers, providers). The contractor and SAMHSA recognize that different audiences are best reached by different types of report formats. For example, reports to Congress will require materials that are concise but offer policy-relevant recommendations. Reports created for SAMHSA Centers and the CORs will require more in-depth information, such as substantive background and discussion sections, to supplement the analytic approach. Reports created for SPF SIG grantees will be concise handouts with helpful and easy-to-read graphics on performance data rather than lengthy text. The assortment of disseminations products developed using the PFS MRT data will include short and long analytic reports, congressional briefings, annual evaluation reports, research and policy briefs, ad hoc analytic reports, journal articles, best practice summaries, and conference or other presentations. Certain dissemination products may be made available on SAMHSA’s website.
The expiration date for OMB approval will be displayed.
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.
1 The “factors” in this question will be assessed with the Community-Level Instrument, previously approved under OMB No. 0930-0279, expiration 02/28/2017.
2 Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
3 West, B. T., Welch, K. B., & Galecki, A. T. (2007). Linear mixed models: A practical guide using statistical software. Boca Raton, FL: Chapman & Hall/CRC.
4 Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester, England: John Wiley & Sons.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | RMC Basic Report & Proposal Template |
| Author | RMC Research Corporation |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |