DIVISION of STATE PROGRAMS MANAGEMENT & REPORTING TOOL
SUPPORTING STATEMENT
The Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Substance Abuse Prevention (CSAP) is seeking approval for a revision, from the Office of Management and Budget (OMB), to the previously approved instrument – Division of State Programs’- Management and Reporting Tool (OMB No. 0930–0354), which expires on July 31, 2020. The instrument in this current package has been renamed as the Division of State Program--Management and Reporting Tool (DSP-MRT). Specifically, CSAP is requesting approval to add program specific questions for two new grant programs—SPF-Rx and Prescription Drug Overdose to the existing OMB approved instrument. Both new programs were funded in FY2016 and require grants management monitoring and GPRA reporting to Congress. The tool includes a standard set of questions used for all programs within the Division of State Programs as well as a specific subset of questions for three individual programs: Partnerships for Success (currently OMB approved), Rx, and Prescription Drug Overdose. The DSP-MRT will primarily gather data related to implementation of the Strategic Prevention Framework (SPF)--Assessment, Capacity, Planning, Implementation, and Evaluation--in the context of preventing underage drinking and/or opioid misuse. SAMHSA’s opioid and underage drinking prevention programs are authorized under Section 516 of the Public Health Service Act.
Background.
Over the past decade, a large number of evaluation studies demonstrated that prevention interventions effectively reduce substance use, as well as delinquent behaviors; violence; and other mental, emotional, and behavioral health problems (e.g., Calear & Christensen, 2010; Lemstra et al., 2010; Ttofi & Farrington, 2011). Among 12- to 20-year-olds from 2002 to 2012, rates of current alcohol use decreased from 28.8% to 24.3%, rates of binge drinking declined from 19.3% to 15.3%, and heavy alcohol use declined from 6.2% to 4.3% (SAMHSA, 2013b). Despite these successes, Underage Drinking (UAD) continues to be a significant public health problem. The 2016 National Survey on Drug Use and Health (NSDUH) report estimates that Approximately 7.7 million underage people reported current use of alcohol, 5.1 million reported binge drinking, and 1.3 million reported heavy alcohol use (SAMHSA, 2015). UAD causes serious harm to the adolescent drinker as well as to the community as a whole (Office of Juvenile Justice and Delinquency Prevention, 2012). Alcohol use by adolescents negatively effects brain development, results in other serious health consequences (e.g., alcohol poisoning, risky sexual behaviors, and addiction), and leads to safety consequences from driving under the influence, poisonings, and other injuries. UAD places youth at increased risk for violence and victimization along with social or emotional consequences (e.g., low self-esteem, depression, anxiety, lack of self-control, stigmatization by peers), academic consequences (e.g., poor academic performance, truancy, suspension or expulsion from school), and family consequences (e.g., poor relationships with parents).
Adolescent drinking can also impose economic consequences, ranging from personal costs (e.g., payment for alcohol treatment or medical services) to familial costs (e.g., parents taking time off of work to drive children to treatment) to community costs (e.g., providing enforcement, supervision, or treatment to underage drinkers). Sacks et al. (2013) estimated that in 2006, UAD was responsible for $24.6 billion (11%) of the total cost to society of excessive alcohol consumption in the United States.
Based on combined 2012–2014 NSDUH data, an annual average of 5.5 million people aged 12 to 20 in the U.S. engaged in binge drinking in the past month. Nationally, 14.44 percent of
all people aged 12 to 20 engaged in binge drinking in the past month (SAMHSA 2015). Prescription Drug Misuse (PDM) refers to the use of illicit drugs to treat pain, attention deficit disorder, or anxiety without a prescription; in a way other than prescribed; or because of the feelings it may elicit (National Institutes of Health, 2011). Regarding prescription drug misuse among people aged 12 or older: 3.8 million misused pain relievers, ~1.9 million misused tranquilizers, ~1.7 million misused stimulants, and ~446,000 misused sedatives (NSDUH 2015).
In 2004, SAMHSA began funding programs using the Strategic Prevention Framework (SPF) to help States, jurisdictions/territories, and tribal organizations implement activities with the goals of preventing the onset and reducing the progression of substance abuse, reducing problems related to substance abuse, and building capacity and infrastructure for prevention. The SPF model consists of five steps: (1) needs assessment; (2) capacity building; (3) strategic planning; (4) implementation of programs, policies, and practices; and (5) evaluation. Grantees also considered cultural competence and sustainability at each step in the process. In 2004, the SPF was used in the State Incentive Grants and then Partnerships for Success in 2011. Currently, the SPF model is being used in both SIG and PFS programs and is the model to be used in two new CSAP programs--SFP-Rx and Prescription Drug Monitoring Programs.
A.2. Purpose and Use of Information
The data on SPF model will allow SAMHSA project officers to systematically collect data to monitor their grant program’s performance along with grantee technical assistance needs. In addition to assessing activities related to the SPF steps, the performance monitoring instruments covered in this statement collect data to assess the following grantee required specific performance measures:
Number of training and technical assistance activities per funded community provided by the grantee to support communities
Reach of training and technical assistance activities (numbers served) provided by the grantee
Percentage of subrecipient communities that submit data to the grantee data system
Number of sub-recipient communities that improved on one or more targeted National Outcome Measures (NOMS)
Number of grantees who integrate Prescription Drug Monitoring Program (PDMP) data into their program needs assessment
Number of naloxone toolkits distributed
The performance monitoring instrument development process included input from grantee-level evaluators, SAMHSA management and CSAP Project Officers, and other stakeholders (see the statistical consultants list in SS-B). After careful review with stakeholders and grantees, revisions were made to streamline the instruments, decrease verbosity, create consistency in assessing infrastructure at the grantee and community subrecipient levels, and address gaps such as those related to leveraged funding. In order to reduce burden in other data collections at SAMHSA, information collected through the DSP MRT, covered by this statement, will also provide data to the national cross-site evaluations conducted within CSAP.
The DSP-MRT is a tool that enables SAMHSA Project Officers to monitor grantees through the SPF process. The MRT gathers all information through a web-based data collection system that uses clickable radio buttons, check boxes, drop-down choice items, and open-ended text boxes, as relevant. It also allows grantees to upload required documents requested by their Project Officers. This web-based data collection instrument is usually completed by the grantee Project Director once each quarter or biannually, depending on the program. The instrument will gather data related to implementation of the SPF steps (Assessment, Capacity, Planning, Implementation, and Evaluation) along with how Health Disparities are addressed through each step. Please see full instrument in attachment 1-3. Clicking the link for each step or section will direct the user to the relevant landing page. For example, the “Assessment” link will direct user to the Assessment landing page.
Data collected will include information on accomplishments and barriers for each step. The Capacity section collects information on workgroup membership and meetings to assess leveraging of partnerships; grantee-level funding and in kind resources to assess leveraging of funds from various sources; and training received by grantees and provided to subrecipients by grantees including training topics, numbers reached, delivery sources, and unfulfilled training needs. The Planning section allows grantees to upload their strategic plans as those become available. The Implementation section requests grantees to provide information on the progress of each of their community subrecipients. The Evaluation section allows grantees to upload their evaluation plans or local evaluation reports as those become available. The Health Disparities section SAMHSA project officers to monitor grantee efforts to fulfill requirements related to SAMHSA’s mission that grantees address health disparities related to substance abuse risks, prevalence, and outcomes. This section allows grantees to upload required health disparities impact statements (plans for how they will address health disparities efforts) as well as describe health disparities-related activities, accomplishments and barriers relevant to each one of the SPF steps.
As aforementioned, DSP-MRT is a revision to the existing Partnerships for Success-MRT. Changes to this instrument include the following:
Inclusion of Community Outcome (Attachment #4)
Inclusion of Evidence Based Practices (3 questions at the end of the standard DSP-MRT)
Both items listed above are needed for GPRA reporting in SAMHSA’s Congressional Justification.
A.3. Use of Information Technology
Grantee staff will provide information in the DSP-MRT through an online data collection system. Using a Web instrument allows for automated data checks as well as for skip procedures and prepopulated fields based on prior responses to certain questions. This will reduce the burden among respondents and data entry error, thereby increasing the efficiency of data entry and improving data quality. The automated data checks will ensure that responses follow the expected format (e.g. numbers or dates where those are expected). Similarly, once completed initially, some items are automatically pre-populated, such as when Grantees provide measure description information on baseline community outcomes data and then only need to change the time frame and outcomes values at later time points.
The Web-based system also allows SAMHSA CSAP Project Officers to review submissions conveniently, request revisions as needed, and then provide approvals to grantees on their submissions as relevant.
A dashboard and other reports will also be available to SAMHSA and the contracting team, as well as the grantees and subrecipients who submit data, so that they can monitor the overall status of data collection and monitor performance. Grantees will have access to their own data.
The Web-based system also allows grantees and SAMHSA Project Officers easy access to a sharepoint site, which contains data submission manuals and other relevant documents, a section with responses to frequently asked questions, and a link to a Technical Assistance Submission form. Grantees and Project Officers can also request technical assistance on their data entry through e-mail and a phone request system. All technical assistance requests are routed to one electronic system which keeps track of requests, follow-ups, and resolutions.
This monitoring tool is collecting information unique to the DSP program grantees that is otherwise not available to project officers. A literature review prepared by the evaluation team in November 2013 confirmed that the information being collected cannot be obtained through other sources. In addition, this data collection was crosswalked with similar instruments across SAMHSA.
Participation in this data collection will not impose a significant impact on small entities. Grantees will usually consist of State agencies, tribal organizations and other jurisdictions. Some subrecipients may be small entities; however, the System for the DSP-MRT is designed to include only the most pertinent information needed to be able to monitor the grantee’s progress and to carry out the evaluation effectively, and their impact will not be significant.
The multiple data collection points for the DSP-MRT are necessary to track and monitor grantees’ and community subrecipients’ progress and change over time. In addition to performance monitoring purposes, SAMHSA will use the data for the purposes of evaluation, and grantee and subrecipient communities will use these data to track their ongoing implementation. Less frequent reporting will affect SAMHSA’s and the grantees’ ability to do so effectively. For example, SAMHSA’s federal requirements require them to report on performance and GPRA measures once each year. New federal health disparities priorities require periodic reports of the activities used to address those priorities.
SAMHSA has made every effort to ensure that data are collected only when necessary and that extraneous collection will not be conducted. For example, grantees report only outcomes required for GPRA measures on an annual basis.
This information collection fully complies with the guidelines in 5 CFR 1320.5(d)(2).
The notice required by 5 CFR 1320.8(d) was published in the Federal Register on May 21, 2018 (83 FR 23475).
These program monitoring tools were based on the original narrative tools completed by previous grant programs. In addition, the tools were reviewed by SAMHSA staff and contractors. These experts provided feedback on each of the data collection instruments and the instruments were revised based on their feedback. Revisions ranged from changes in the instructions to simplify them, to the addition of a module on health disparities in the monitoring progress report. See Supporting Statement B for the list of individuals consulted throughout the development process of the instruments.
No cash incentives or gifts will be given to respondents.
The DSP-MRT only requests personal data through the Contact Information section of the system. That staff role, name, e-mail, and telephone number data collected through that instrument are collected to allow contract staff to provide grantee and subrecipient login information for the system, and to facilitate contact with the grantee and subrecipient staff on their data entry, data cleaning needs, and technical assistance requests. This identifying information will be accessible only to select contractor evaluation staff and Project Officers at SAMHSA. No other personal information will be collected from respondents as the focus of the data collection is on the programmatic characteristics of the grantees and subrecipients.
No individual-level or personal data will be collected through the system. Grantee staff will provide information about their organizations and their activities, rather than information about themselves personally. The instruments collect programmatic data at the grantee and community levels along with aggregated, non-identifying individual-level data (e.g., community outcomes data). Sensitive respondent information, such as birthdates and Social Security Numbers, will not be collected.
The contracting team takes responsibility for ensuring that the Web and data system is properly maintained and monitored. Server staff will follow standard procedures for applying security patches and conducting routine maintenance for system updates. Data will be stored on a password-protected server, and access to data in the system will be handled by a hierarchy of user roles, with each role conferring only the minimum access to system data needed to perform the necessary functions of the role.
While not collecting individual-level data, contractor staff are trained on the importance of privacy and in handling sensitive data.
There are no questions of a sensitive nature in this collection.
The number of data collection respondents will vary by year because of the varying lengths in grants, data collection time points, and each cohort’s grant end dates. As such, the burden and respondent cost will also vary by year.
All programs within DSP, and all future cohorts, are expected to complete their monitoring reports between two to four times per year, depending on the grant requirements of the program. The DSP Management Reporting Tool is estimated to take 3 hours to complete per response; this includes time to look up and compile information (2.5 hours) and time to complete the Web-instrument (1.5 hour). The estimated burden time is based on test instruments completed by evaluation staff members that have experience working with grantees (see Section B.4 for more detail) as well as grantees who participated in a pilot of the instrument. There are no direct costs to respondents other than their time to complete the instrument. Table below provides the details of the annual burden for the DSP-MRT, which also includes section A-C for program specific questions. The estimate for each program specific section is 1 hour.
Burden Table: FY2018—FY2020 Annualized Burden
Instrument |
Number of Respondents |
Responses per Respondent |
Total Number of Responses |
Hours per Response |
Total Burden Hours |
Average Hourly Wage |
Total Respondent Costa |
DSP -MRT |
117 |
4 |
468 |
3 |
1,404 |
$44.19 |
$ 62,042 |
Section A: Rx |
25 |
4 |
100 |
1 |
100 |
$44.19 |
$4,419 |
Section B: PDO |
23 |
3 |
69 |
1 |
69 |
$44.19 |
$3,049 |
Section C: PFS Outcomes |
71 |
1 |
71 |
3 |
213 |
$44.19 |
$9,412 |
FY2018-FY2020 Total |
117 |
|
708 |
|
1,786 |
|
$ 78,922 |
a Total respondent cost is calculated as total burden hours x average hourly wage.
There are no respondent costs for capital or start-up or for operation or maintenance.
The total estimated cost to the government for the data collection from FY 2018 through FY 2020 is $3,086,153. This includes approximately $2,602,547 for developing the instruments; programming and maintaining the online data collection system; providing data collection training to grantees and subrecipients; processing, cleaning, and housing data; and analyzing and reporting data. Approximately $55,602 per year represents SAMHSA costs to manage/administer the data collection and analysis for 25% each of two employees (GS-14-10, $111,203 annual salary). Approximately $105,600 per year represents SAMHSA costs to monitor and approve grantee reporting in these instruments (10% time of 10 Project Officers at $105,600 annual salary). The annualized cost is approximately $1,028,717.
Currently there are 1,816 burden hours in the OMB inventory. The program is requesting 1,786 hours, a decrease of 30 hours. Even though Section C added 213 burden hours, the decrease in the number of respondents reduced the overall burden for the standard tool.
A.16. Time Schedule, Publications, and Analysis Plan
Time Schedule for Data Collection
Activity |
Time Schedule |
Prepare for data collection, including programming Web system |
Jan. 2018-Oct. 2018 |
Obtain OMB approval for data collection |
Oct. 2018 |
Collect data |
Oct. 2018–September 2020 |
Analyze data --Quantitative data submitted through the biannual annual progress report |
April 2018–September 2021 |
Disseminate of findings |
Ongoing for monitoring purposes. |
The data from the DSP-MRT will primarily be used by SAMHSA Project Officers to monitor the progress of their grantees. However, data from the monitoring reports will also be used for evaluation purposes, as the process data may inform specific outcomes. For either purpose, the objective for all reports and dissemination products is to provide user-friendly documents and presentations that help SAMHSA successfully disseminate and explain the findings to a variety of target audiences. Audiences for these reports will include Congress, the ONDCP, SAMHSA Contracting Officer’s Representatives (CORs), grantees, and the broader substance abuse prevention field (e.g., academia, researchers, policy-makers, providers). SAMHSA recognizes that different audiences are best reached by different types of report formats. For example, reports to Congress and ONDCP will require materials that are concise but offer policy-relevant recommendations. Reports created for SAMHSA Centers and the CORs will require more in-depth information, such as substantive background and discussion sections, to supplement the analytic approach. Reports created for grantees will be concise handouts with helpful and easy-to-read graphics on performance data rather than lengthy text. The assortment of disseminations products developed using the data will include short and long analytic reports, congressional briefings, annual evaluation reports, research and policy briefs, ad hoc analytic reports, journal articles, best practice summaries, and conference or other presentations.
The DSP-MRT uses a series of interdependent analysis frameworks that have been selected to maximize the coverage of key objectives of the SPF in the prevention of onset and the reduction of the progression of UAD and PDM and their consequences. PFS communities may select additional outcomes that are specific to their community (e.g. heroin). Monitoring data will be collected through the web site. Data will be used to report to Congress regarding the GPRA as specified in the SAMHSA Annual Justifications of Budget Estimates as well as for grants monitoring purposes. Data may be used in different evaluation studies for the purpose of providing contextual information to more specific outcome data.
Qualitative analyses of the monitoring data focus primarily on open ended responses grantees provide to describe their SPF step accomplishments and barriers. Preparation for coding will include developing a dictionary or codebook in which codes will be carefully defined and logged so that coders are able to follow their meaning and know when to apply the codes to text within an interview. Codes will reflect prominent themes relevant to interpreting evaluation findings. To ensure reliability in the coding process, coders will then be assigned to work independently and concurrently on a subset of the open-ended response data. A kappa coefficient of .8 or higher will be maintained on all codes. Any discrepancies will be resolved between coders to ensure consistent application of codes. Upon completion of coding, the findings will be compiled on the basis of the prominence of codes (or themes) and organized around the major research questions and constructs. The findings that emerge will be used to examine grantee progress through the SPF steps.
A.17 Display of Expiration Date
OMB approval expirations dates will be displayed.
A.18. Exceptions to Certification for Statement
There are no exceptions to the certification statement. The certifications are included in this submission
REFERENCES
Calear, A. L., & Christensen, H. (2010). Systematic review of school-based prevention and early intervention programs for depression. Journal of Adolescence, 33(3), 429–438.
Lemstra, M., Bennett, N., Nannapaneni, U., Neudorf, C., Warren, L., Kershaw, T., et al. (2010). A systematic review of school-based marijuana and alcohol prevention programs targeting adolescents aged 10–15. Addiction Research and Theory, 18, 84–96.
National Institutes of Health. (2011, October). Prescription drugs: Abuse and addiction (NIH Publication No. 11-4881). Retrieved from http://www.drugabuse.gov/sites/default/files/rrprescription.pdf
Office of Juvenile Justice and Delinquency Prevention. (2012, September). Effects and consequences of underage drinking (NCJ 237145). Juvenile Justice Bulletin. Washington, DC: Author. Retrieved from http://www.ojjdp.gov/pubs/237145.pdf
Office of National Drug Control Policy. (2011). Epidemic: responding to America’s prescription drug abuse crisis. Retrieved from: http://www.whitehouse.gov/sites/default/files/ondcp/issues-content/prescription-drugs/rx_abuse_plan_0.pdf
Office of the Surgeon General. (2007). The Surgeon General’s call to action to prevent and reduce underage drinking. Rockville, MD: Author. Retrieved from: http://www.surgeongeneral.gov/library/calls/underagedrinking/calltoaction.pdf
Sacks, J. J., Roeber, J., Bouchery, E. E., Gonzales, K., Chaloupka, F. J., & Brewer, R. D. (2013). State costs of excessive alcohol consumption, 2006. American Journal of Preventive Medicine, 45(4), 474–485.
Substance Abuse and Mental Health Services Administration (SAMHSA). 2015 Key Substance Use and Mental Health Indicators in the United States: Results from the 2015 National Survey on Drug Use and Health. Figure 24. Rockville, MD: SAMHSA, 2016. Available at :http://www.samhsa.gov/data/sites/default/files/NSDUH-FFR1-2015/NSDUH-FFR1
Ttofi, M. M., & Farrington, D. P. (2011). Effectiveness of school-based programs to reduce bullying: A systematic and meta-analytic review. Journal of Experimental Criminology, 7, 27–56.
United States Department of Health and Human Services. (2011). National prevention strategy: America’s plan for better health and wellness. Washington, DC: National Prevention Council. Retrieved from: http://www.surgeongeneral.gov/initiatives/prevention/strategy/report.pdf
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Snaauw, Roxanne |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |