Program Evaluation for Prevention Contract Evaluation Activities
Supporting Statement
The Substance Abuse and Mental Health Services Administration’s (SAMHSA), Center for Behavioral Health Statistics and Quality (CBHSQ) is requesting approval from the Office of Management and Budget (OMB) for a revision to the data collection activities related to the cross-site evaluation of SAMHSA’s Strategic Prevention Framework for Prescription Drugs (SPF-Rx) - OMB No. 0930-0377, expiration date August, 31, 2020. SAMHSA funds the Program Evaluation for Prevention Contract (PEPC), which supports the cross-site evaluation activities for SPF-Rx.
SAMHSA requests approval for the following data collection tools:
Exhibit 1. Data Collection Tools
Instrument |
OMB Request |
Attachment |
Annual Implementation Instrument (survey instrument) |
Revision |
1 |
Grantee-Level Outcomes Module (secondary data collection instrument) |
Revision |
2 |
Community-Level Outcomes Module (secondary data collection instrument) |
Revision |
3 |
Grantee-Level Interview (questionnaire) |
Revision |
4 |
Substitute Data Source Request (form) |
Removal |
|
The grant program is summarized below.
The SPF-Rx grant program is designed to address nonmedical use of prescription drugs as well as opioid overdoses by raising awareness about the dangers of sharing medications and by working with pharmaceutical and medical communities on the risks of overprescribing. At the end of FY 2016, SAMHSA awarded the 5-year SPF-Rx grant to 21 states and 4 tribal organizations. The SPF-Rx evaluation assesses 1) program implementation, 2) outcomes, and 3) the barriers and facilitators affecting each grantee.
Grantees use SAMHSA’s Strategic Prevention Framework (SPF) to plan, implement, and evaluate their prevention projects. The SPF, which comprises five steps and two guiding principles, provides a comprehensive process for focusing communities’ prevention initiatives on the most pressing needs and priority substance use and related health problems.
Use of the SPF ensures that prevention efforts are data-driven, dynamic (i.e., involve continuous needs assessment and adjustment of prevention strategies, as needed), engage diverse state, tribal, and community partners, and focus on population-level change. The SPF-Rx program has several important requirements: grantees must work collaboratively with other state, tribal, and community stakeholders (e.g., schools; businesses; law enforcement; pharmaceutical and medical communities; youth, young adults, and other community members including parents) to achieve their objectives. Most, though not all, grantees choose to fund subrecipient organizations to implement interventions at the community level. Grantees must also design and implement their own local evaluations and participate in SAMHSA’s cross-site evaluation.
Opioid misuse, opioid use disorder, and opioid overdose are significant public health issues in the United States (U.S.). In the most recent survey, SAMHSA’s National Survey on Drug Use and Health (NSDUH) estimated that 3.6 percent of respondents reported misuse of prescription pain relievers in the past year (SAMHSA, 2019). Since 2000, the rate of deaths from drug overdoses has increased 137 percent, including a 200 percent increase in the rate of overdose deaths involving opioids (Rudd, Aleshire, Zibbell, & Gladden, 2016). Between 2003 and 2012, more deaths have been due to opioid analgesic overdoses than to heroin and cocaine combined (Centers for Disease Control and Prevention, 2012). Drug-related overdose is currently the nation’s leading cause of accidental death, with deaths from opioid overdose playing a significant role in this increase (Hedegaard, Minino, & Warner, 2018). Opioid overdose fatalities can be attributed to both prescription medications, such as morphine, codeine, oxycodone, and others, and to illegal drugs such as heroin and illegally manufactured fentanyl and its analogs. Beginning in late 2013, states began to see a spike in overdose deaths due to fentanyl and its analogs (Centers for Disease Control and Prevention, 2015). In response to these crises, the SPF-Rx grant program seeks to provide infrastructure for states, tribal entities, and their subrecipients to address issues of prescription drug misuse.
The SPF-Rx evaluation will assist SAMHSA to better understand whether the SPF-Rx program impacts opioid use at the state, tribal, and subrecipient levels, and which programmatic factors are contributing to that change. The tools included in this package allow researchers to collect process and outcome data to better understand the program. The Public Health Services Act requires SAMHSA to monitor program performance and document the impact of government funding. The SPF-Rx data collection tools are therefore necessary for proper performance of SAMHSA’s required agency functions of program oversight.
Data collected through the tools described in this statement will be used for the national cross-site evaluation of SAMHSA’s SPF-Rx program. This request for revision covers continued data collection through FY 2023, as the evaluation is expected to continue until that date. The PEPC team will systematically collect and maintain Annual Implementation Instrument (AII) and Grantee- and Community-level Outcomes data submitted by SPF-Rx grantees through the online PEPC Data Management System (DMS). The evaluation also includes qualitative interviews with SPF-Rx grantee project directors using the Grantee-Level Interview tool.
The following primary evaluation questions (EQs) guide the cross-site evaluation:
EQ1: Was the implementation of SPF-Rx associated with desired proximal and distal outcomes, including safer opioid prescribing practices and decreases in prescription drug misuse and opioid overdoses?
EQ2: How did SPF-Rx grantees use Prescription Drug Monitoring Programs (PDMP) to improve proximal and distal outcomes?
EQ3: What barriers and facilitators affected SPF-Rx implementation and outcomes (e.g., characteristics of partnerships, concentration of effort, infrastructure, laws and regulations, state and community contextual factors), and how did grantees address these barriers?
The evaluation consists of yearly data collection of the AII and of the Grantee- and Community-Level Outcomes Modules. The AII contains questions related to: grantee and community needs assessment, data sources, resources, capacity building and sustainability; interventions that have been implemented; and the targeted population and reach. The AII will serve as the source of both process and implementation data as well as provide independent variables for the evaluation of outcomes. Grantee- and Community-Level outcomes are the dependent variables and were selected as indicators of reductions in opioid misuse. The outcomes modules include four categories of indicators: opioid overdose morbidity and mortality; prescription drug monitoring; prescriber use of PDMPs; and consumption. The Grantee-Level Interview seeks to answer EQ3 in more detail; describing the barriers and facilitators to SPF-Rx implementation and outcomes. This qualitative interview was conducted during Year 1 and 2 of the evaluation with all grantees and will be conducted again during Year 4 of the evaluation.
Exhibit 2 provides an overview of the data collection method, frequency of data collection, and number of times each tool is collected for the SPF-Rx data collection instruments.
Exhibit 2. SPF-Rx Cross-Site Evaluation Data Collection Tools (N=25 grantees and 123 subrecipients)
Instrument |
Data Collection Method |
Frequency of Data Collection |
Maximum Number of Data Collections |
Attachment Number |
Annual Implementation Instrument |
Grantees and subrecipients both submit in the DMS |
Yearly (subrecipients complete for each year funded) |
3 times: Years 4-6 |
1 |
Grantee-Level Outcomes Module |
Grantees submit in the DMS |
Yearly |
3 times: Years 4-6 |
2 |
Community-Level Outcomes Module |
Grantees submit for subrecipient communities in DMS |
Yearly |
3 times: Years 4-6 |
3 |
Grantee-Level Interview |
Grantee interviews are recorded and transcribed; stored on secure drive |
Baseline and follow-up |
2 times: One time under this OMB approval in Year 4 |
4 |
SAMHSA’s SPF-Rx program is designed with the premise that changes at the community level will lead to measurable changes in substance use and misuse at the state and tribal levels. It assumes that effective state, tribal, and community change requires comprehensive efforts targeting youth and adults, as well as the environments in which they live.
The goal of SAMHSA’s SPF-Rx cross-site evaluation is to provide data on activities and services that were delivered; program participants; procedures, infrastructure supports that facilitate program implementation; implementation barriers; program outcomes and impacts; and the extent to which grantees were prepared or able to sustain their programs at the end of the grant period. In addition, SAMHSA’s qualitative interviews and site visits examine factors in the state, tribal, and community environments (i.e., context) that influence program implementation and outcomes.
The theory of change guiding the SPF-Rx program is that well implemented prevention efforts at the community-level will result in population-level change. This includes reduced prescription drug misuse and continued enhancements to state, tribal, and community prevention systems. This section describes the practical utility of the SPF-Rx data collection. Since the SPF-Rx cross-site evaluation has already commenced, this section also reviews how SAMHSA is using the SPF-Rx data.
The SPF-Rx evaluation is designed to objectively and rigorously measure population-level changes in prescription drug use and its impact (e.g., opioid overdose morbidity and mortality), and describe conditions and changes in state, tribal, and community prevention systems (e.g., PDMP use). In the first two years of the cross-site evaluation, SPF-Rx data provided baseline outcomes data and provided SAMHSA with a better understanding of the myriad of evidence-based interventions that are being implemented at the state, tribal, and community levels and their impact on opioid-related outcomes.
Instrumentation
The SPF-Rx data collection efforts include the AII, Grantee- and Community-Level Outcomes Modules, and Grantee-Level Interview. These cross-site measures provide process data regarding: progression through the SPF model; challenges and successes experienced during these steps; PDMP infrastructure; interventions implementation; prescriber use and prescribing patterns and outcomes; training and technical assistance (T/TA); and funding. This data collection emphasizes the SPF-Rx impact on outcomes related to the prevalence of prescription drug misuse, and capacity for and use of PDMP for monitoring prescriber behavior and prevention purposes. Data collection for opioid related overdose events and deaths have been removed from the SPF-Rx data collection efforts as they are collected through SPARS and thus duplicative. The emergence of prescription drug misuse as a serious public health issue highlights the critical need for the SPF-Rx cross-site evaluation to examine the implementation and effectiveness of prevention interventions developed to target this issue.
The AII, Grantee- and Community-Level Outcomes Modules, and Grantee-Level Interviews are used to collect data to measure the main constructs of interest in order to answer the SPF-Rx EQs. The instrumentation is described in detail below.
Annual Implementation Instrument (AII, Attachment 1): The AII is a survey instrument collected through PEPC’s DMS. It is designed to be completed by grantees and subrecipient community project directors. The PEPC evaluation team collects AII data yearly to monitor state, tribal entity, and community-level performance, and to evaluate the effectiveness of the SPF-Rx program across states, tribal entities, and subrecipient communities. The AII provides process data related to funding use and effectiveness, organizational capacity, collaboration with community partners, data infrastructure, planned intervention targets, intervention implementation (e.g., categorization, timing, dosage, and reach), evaluation, contextual factors, T/TA needs, and sustainability. Repeated collection of these data is needed to: 1) track the grantees and subrecipients’ progress and changes to the aforementioned indicators over time; and 2) allow SAMHSA and the grantees to monitor performance and ongoing implementation.
The AII included in this data collection includes minor revisions from the previously approved data collection. The evaluation team revised the instructions and definitions for some of the variables to reduce confusion and improve clarity for the respondents.
The AII will be collected 3 times under this data collection; in Year 4 (FY 2021), Year 5 (FY 2022), and Year 6 (FY 2023). In Years 4 and 5, the evaluation team will collect data for the last 2 years of the grant program, and in Year 6, data will only be collected for grantees and subrecipients receiving no-cost extensions. The estimated burden for the AII is 4 hours per respondent, including time to gather relevant information and enter it into the Data Management System (DMS). The previous data collection estimated the AII burden as 2.3 hours per respondent. However, those estimates preceded data collection. The current burden reflects the average of the actual burden estimates provided by 6 current SPF-Rx grantees.
Grantee- (Attachment 2) and Community-Level Outcomes Modules (Attachment 3):
The Grantee- and Community-Level Outcomes Modules are survey instruments collected through PEPC’s DMS. Grantees use the Outcomes Modules instruments to provide outcome data about consumption, consequence, and risk and protective factors that contribute to substance use and misuse. They provide annual outcome data for opioid prescribing practices and prescribers’ use of PDMPs (from PDMP data) at both the grantee, subrecipient, and community levels. Outcomes data on overdose events, deaths, as well as opioid consumption will be collected separately through the SAMHSA Performance Accountability and Reporting System (SPARS) and existing survey data and have therefore been removed from these modules.
Based on the experience of 6 current SPF-Rx grantees we queried, the estimated burden for the Grantee-Level Outcomes Modules was 6.2 hours, an increase of 3.2 hours from the previous OMB statement estimate of 3 hours. This estimates was based on the full instrument submitted with the 2017 OMB prior to data collection. This estimate was derived from both the time needed to gather the relevant information for completing the modules and the time it takes to enter the data into the DMS. The estimated burden for the revised/reduced Grantee-Level Outcomes Module submitted with this package is 2.5 hours. Grantee-Level Outcomes Module burden included in this data collection is a greater reduction than what appears based on comparison with the 2017 data collection, as it is based on the actual burden of 6.2 hours as reported by the 6 grantees queried. Therefore, the burden estimate for the Grantee-Level Outcomes Module submitted with this package was reduced by 60% to 2.5 hours.
Based on the experience of the 6 current SPF-Rx grantees, the estimated burden for the Community-Level Outcomes Module is 3.5 hours, an increase of 30 minutes from the previous OMB statement of 3 hours. This estimate was based on the full instrument submitted with the 2017 OMB prior to data collection. This estimation is derived from both the time needed to gather the relevant information for completing the modules and the time it takes to enter the data into the DMS. The estimated burden for the revised/reduced Community-Level Outcomes Module submitted with this package is 1.25 hours.
In summary, the current burden estimate is 2.5 hours for the Grantee Level Outcomes module and 1.25 hours for the Community Level Outcomes module; reducing the burden from the original OMB approval.
The revised Community-Level Outcomes Module is less burdensome because it includes fewer measures than the Grantee-Level Module. The new process involves grantees selecting one of their subrecipients from a dropdown menu. For a new outcome, grantees click on the “Add a Record” button. Once they have added records, they are able to view previously added records for the selected subrecipient. This reduces burden on this instrument through two processes:
Grantees can copy background information for a given outcome from one subrecipient to another, so that grantees only need to provide the subrecipient-specific outcome data.
After the initial data entry for a subrecipient, grantees only need to provide data on the follow-up period using the “Add Follow-Up Data” link provided on the page.
The Grantee and Community Level Outcomes Module instruments included in this package have been revised from the previously approved 2017 data collection. The following questions (Exhibit 3) were removed as they were found to be duplicative of modules that exist in SAMHSA Performance Accountability and Reporting System (SPARS) for other grant programs. To streamline reporting, SPF-Rx grantees will complete the modules in SPARS instead.
Exhibit 3: Removal Questions from Outcomes Modules.
Question to be removed from Grantee and Community Level Outcomes Module |
Rationale for removal |
1.1 Hospital data for Opioid Overdoses |
Duplicative of modules available in SPARS |
1.2 Other Opioid Overdose Events |
Duplicative of modules available in SPARS |
1.3 Opioid Overdose Deaths |
Duplicative of modules available in SPARS |
3 Consumption: Survey Estimates of Prescription Drug Misuse |
Duplicative of modules available in SPARS (Community) and the National Survey on Drug Use and Health (Grantee) |
Substitute Data Source Request Form: The Substitute Data Source Request form has been removed because the same module exists in SPARS and is therefore duplicative.
Grantee-Level Interview (Attachment 4): The Grantee-Level Interview is a semi-structured interview, conducted by telephone with grantee staff. This instrument is designed to collect more in-depth qualitative information on organizational infrastructure, use of PDMP data, collaboration, funding use and effectiveness, subrecipient selection, criteria for intervention selection, processes to decrease health disparities, and evaluation activities. The Grantee-Level Interview was conducted at the beginning of the grant and will occur again in the final year of the grant. Collecting baseline and follow-up data is necessary to assess the grantees’ progress and change in outcomes over the course of the grant. Due to the timing of OMB approval and implementation of the interview, the baseline data collection occurred at the end the second year and beginning of the third year of the SPF-Rx grants, requiring grantees to provide retrospective information. Follow-up data will be collected during the final year of the SPF-Rx grant program (Year 4 of the evaluation).
The SPF-Rx cross-site evaluation is expected to have important program and policy implications at the federal, state, tribal, and community levels. It will provide valuable information to the prevention field about best practices in real-world settings, including what types of interventions should be funded and implemented to reduce prescription drug misuse. Additionally, the evaluation will provide information about ways to build PDMP capacity at the state, tribal, and community levels. The evaluation is ultimately designed to use data to inform resource allocation and programming to prevent prescription drug misuse.
Changes: See Attachment 5
All efforts have been made to minimize respondent burden, while obtaining the essential information needed to answer the EQs. The use of web-based data submission methods decreases respondent burden as compared to that required for alternate methods, such as a paper format, by allowing direct transmission of the data. During the data collection period respondents can enter and submit the data at a time and location that is convenient for them. In addition, the data entry and quality control mechanisms built into the web-based portal reduce errors that might otherwise require follow-up, thus reducing burden compared to that required for hardcopy data collection. Whenever possible, the PEPC team uses automated electronics to improve data quality and reduce burden for respondents. Additionally, any publicly produced documents will be 508 compliant for accessibility to the public.
SPF-Rx grantee staff will submit 100 percent of their responses electronically by providing their AII and the Grantee- and Community-Level Outcomes data through SAMHSA’s DMS. Use of a web-based system reduces both respondent burden and data entry error, thereby increasing the efficiency of data entry and improving data quality, for the reasons listed below.
The automated data checks will ensure that responses follow the expected format (e.g., numbers or dates where those are expected).
Both grantees and subrecipients will answer questions on prevention interventions that they have implemented. However, subrecipients will also answer questions on their progress through the SPF steps, prevention capacity, and related funding measures (see Exhibit 2 for timing of various data collection items). The DMS provides a different set of questions depending on whether the respondent is a grantee or subrecipient, so grantees will only see questions that are required for them.
Both instruments have automated data checks as well as skip procedures and prepopulated fields based on prior responses to certain questions. Only the questions that are required at that period will appear on the instrumentation.
Web-based systems allow grantees to copy information from one form to another and then change information as needed, such as when they need to provide similar data on the same measures for multiple communities, where only the outcomes value differs.
Web-based systems also allow the cross-site evaluation team to review submissions efficiently, request revisions or clarifications as needed, and then approve grantee submissions as appropriate. This process increases accuracy of data, which ultimately makes these data easier to analyze and strengthens the analysis and results. Grantees have access to their own data by viewing it in the DMS and are able to download their AII data from the DMS.
This evaluation collects information unique to SPF-Rx programs that is otherwise not available to Project Officers or the PEPC cross-site evaluation team. With an eye toward minimizing duplication and burden, the PEPC evaluation team ensured data collected from each instrument is non-duplicative and complementary to the other evaluation components and program monitoring tools. The team identified several outcomes modules related to morbidity and mortality that were available in SPARS and were removed them from the Grantee and Community level outcomes modules included in this statement.
Participation in this evaluation will not impose a significant impact on small entities. SPF-Rx grantees are state agencies, tribal entities, and other jurisdictions. Some subrecipients may be small entities such as local coalitions; however, the data collection instruments have been designed to include only the most pertinent information needed to understand progress and to carry out the evaluation and feasibility study effectively. Burden on small entities is expected to be minimal.
The data collection schedule represents the minimum amount of information needed for the government to accomplish the objectives of its evaluation and to meet data reporting requirements. SAMHSA made every effort to ensure that data are collected only when necessary, and that extraneous collection will not be conducted. For example, the AII tool for SPF-Rx collects grantee and subrecipient implementation data annually, allowing the PEPC team to track implementation progress among grantees and their subrecipients and for regular data feedback to grantees on subrecipient implementation. Timing information can be found in Exhibit 2 for SPF-Rx.
The multiple data collection points for the SPF-Rx cross-site evaluation in the DMS are necessary to track and evaluate progress and change over time for grantees, tribes, and communities. SAMHSA uses these data for the purposes of the cross-site evaluation for the SPF-Rx programs, and grantees use these data to track ongoing implementation of their efforts under this grant. Less frequent reporting could impede SAMHSA’s and the grantees’ ability to do so effectively. For example, SAMHSA is federally required to report on GPRA measures annually. GPRA measures are included in the SPF-Rx cross-site evaluation and therefore must be collected each year from grantees.
AII and Grantee- and Community-Level Outcomes Module
The evaluation team collects AII and Grantee- and Community-Level Outcomes data yearly to track trends across time. Without yearly updates to the AII, we would not have trends data on the type and reach of the SPF-Rx program, including the number of interventions implemented and the numbers reached or served by those interventions. Similarly, the evaluation team uses the outcomes data from the Grantee- and Community-Level Outcomes Modules related to, PDMP use, and prescription drug misuse as the dependent variables to measure the impact of the program over time. Less than yearly data points would inhibit our ability to conduct reliable trends analysis.
Grantee-Level Interview
In the previous data collection, SAMHSA collected the Grantee-Level Interview at three time points; baseline, Year 3, and Final Year. The evaluation team has reduced this to two time points to reduce burden and to only collect data when they are most needed; at the beginning and end of the evaluation.
This information collection fully complies with the guidelines in 5 CFR 1320.5(d)(2).
The notice required by 5 CFR 1320.8(d) was published in the Federal Register on February 24, 2020 (85 FR 10455). No comments were received.
The SPF-Rx evaluation tools were developed by SAMHSA and the PEPC evaluation team. The PEPC evaluation team conducted interviews with 5 grantees before the development phase. Feedback from these interviews was used to inform the SPF-Rx evaluation tool development. Individuals provided feedback on the data collection instruments and the instruments were revised based on their feedback. Revisions ranged from changes in the instructions, to simplifying and streamlining data collected across tools and across overlapping grant programs.
No cash incentives or gifts will be given to respondents.
All members of the PEPC team will receive general awareness training and role-based training commensurate with the responsibilities required to perform the tasks of the project. Prior to performing any project work or accessing any system, and annually thereafter throughout the life of the study, each team member will have completed the SAMHSA Security Awareness Training required by the agency, as well as Records Management and Human Subjects Research Training. The project will maintain a list of all individuals who have completed these trainings and will submit this list to the Project Officer upon request.
The study teams will safeguard the names of respondents, all information or opinions collected in the course of interviews and observations, and any information about respondents learned incidentally during the project. Sensitive respondent information, such as birthdates and Social Security numbers, will not be collected. Although PEPC will not be collecting personally identifiable information (PII), the team is trained on privacy and properly handling sensitive data. Hard copies of evaluation data and notes containing personal identifiers will be kept in locked containers or a locked room when not being used. Every effort will be taken to limit access to data to only those persons who are working on the project and who have been instructed in appropriate Human Subjects requirements for the project. All data, notes, recordings, etc. will be provided to SAMHSA at least 30 days prior to contract end date. SAMHSA will ensure documentation of destruction is completed by the contractor once all information and data is provided to SAMHSA.
The data collection instruments do not request PII. They collect programmatic data at the grantee and community levels, along with aggregated, non-identifying PII (e.g., community outcomes data). Identifying information such as individual names and addresses will not be part of any electronic record. Electronic files and audio files will be accessible only to project staff who have received permission from the Project Director to access them, and files containing data are stored on a platform requiring password protection and additional authentication prior to accessing. Access to network-based data files will be controlled through the use of Access Control Lists or directory- and file-access rights based on user account ID and the associated user group designation. Staff will be instructed on the proper use of PCs for the storage, transfer, and use of sensitive information and the tools available such as encryption.
The PEPC team takes responsibility for ensuring that the web and data systems are properly maintained, monitored, and secured. Server staff will follow standard procedures for applying security patches and conducting routine maintenance for system updates. Data will be stored on a password-protected server, and access to data in the system will be handled by a hierarchy of user roles, with each role conferring only the minimum access to system data needed to perform their specific functions.
Individuals and organizations providing information to the SPF-Rx evaluation will be told the purposes for which the information is collected and that any identifiable information about them will not be used or disclosed for any other purpose. Identifiers such as name, email address, and position will be collected to facilitate survey administration and to notify respondents of the grantee survey in year 4. Once data collection is complete, personal identifiers will be removed from the data and destroyed.
The SPF-Rx cross-site evaluation study was presented to the contractor’s IRB and was found to be exempt from IRB review (Abt Associates IRB #1087). This exemption only applies to the protocols submitted as attachments to this Data collection, and if any of the protocols are changed in the future, the study will be resubmitted to determine whether further IRB review is required.
The information reported by respondents for the SPF-Rx does not ask for sensitive personal information or include questions of a sensitive nature. The focus of the SPF-Rx data collection is on the programmatic characteristics of the SPF-Rx grantees and subrecipient communities. Grantee staff provide information about their organizations and SPF-Rx activities, rather than information about themselves personally.
This section provides annualized and total burden estimates for each SPF-Rx instrument included in this OMB statement. The total burden for this entire OMB statement is 1,867.5 hours and $61,766.12, shown in Exhibit 4.
Exhibit 4. Total Data collection Burden
Study |
Number of Respondents |
Total Responses |
Total Burden Hours a |
Total Wage Cost b |
SPF-Rx Total |
475 |
696 |
1867.5 |
$61,766.83 |
a Total Burden hours includes Grantee PD or Evaluator hourly wage and Subreceipient Staff hourly wage. The Grantee PD or Evaluator hourly wage is based on the mean hourly wage for state government managers, as reported in the 2018 Occupational Employment (OES) by the Bureau of Labor Statistics (BLS) found at https://www.bls.gov/oes/current/naics4_999200.htm#11-00000. Subrecipient Staff hourly wage is based on the mean hourly wage for local government counselors, social workers, and other community and social service specialists, as reported in the 2018 OES by the BLS found at https://www.bls.gov/oes/current/naics4_999300.htm.
b Total wage cost is calculated as total burden hours × average hourly wage by staff type.
For Years 4 and 5 of the SPF-Rx evaluation, the number of data collection responses will be consistent for grantees, but may vary for their subrecipients, based on the timing of their funding. Additionally, the evaluation team will only collect data in Year 6 from grantees and their subrecipients who received no-cost extensions. As such, the burden and respondent cost may vary by year. Exhibit 5 provides an overview of the total estimated annual number of responses for each year and the total burden for the remainder of the evaluation (Years 4-6). Across the SPF-Rx evaluation instruments, the total burden is estimated to be 1,867.5 hours. The total cost burden is estimated to be $61,766.83.
Exhibit 5. SPF-Rx Evaluation Burden Totals by Year
Year |
Number of Respondents |
Total Responses |
Total Burden Hours a |
Total Wage Cost b |
Year 4 |
223 |
321 |
845.75 |
$28,178.30 |
Year 5 |
199 |
297 |
808.25 |
$26,541.83 |
Year 6 |
53 |
78 |
213.50 |
$7,045.99 |
Total |
475 |
696 |
1856.40 |
$ 61,766.12 |
a Total Burden hours includes Grantee PD or Evaluator hourly wage and Subreceipient Staff hourly wage. The Grantee PD or Evaluator hourly wage is based on the mean hourly wage for state government managers, as reported in the 2018 Occupational Employment (OES) by the Bureau of Labor Statistics (BLS) found at https://www.bls.gov/oes/current/naics4_999200.htm#11-00000. Subrecipient Staff hourly wage is based on the mean hourly wage for local government counselors, social workers, and other community and social service specialists, as reported in the 2018 OES by the BLS found at https://www.bls.gov/oes/current/naics4_999300.htm.
b Total wage cost is calculated as total burden hours × average hourly wage by staff type.
Data collection conducted under the previous OMB approval (OMB No. 0930-0377, expiration date August, 31, 2020), Years 1-3, are not included in this burden estimate.
Annual Implementation Instrument
The AII is required annually of all grantees and subrecipients that have been funded during that year. Key assumptions related to the burden include:
In Years 4 and 5, we expect that 25 SPF-Rx grantees and 123 SPF-Rx subrecipients will complete the AII one time each year.
In Year 6, we only anticipate grantees and subrecipients with no-cost extensions will complete the AII. Based on SAMHSA’s experience with similar grant programs where 25-30 percent of grantees request a no-cost extension, we estimate 7 grantees will request a no-cost extension, and therefore the Year 6 annual burden only includes those grantees and their approximately 32 subrecipients.
The AII is estimated to take 4 hours to complete each response period; this includes 2.5 hours to research and compile information, and 1.5 hour to complete the web instrument.
There are no direct costs to respondents other than their time to complete the instrument.
Six grantees were queried to determine the time it took to complete the AII instrument. Those times were averaged to create the burden estimates. Exhibits 5-7 provide detail of the annual burden for the AII for Years 4-6, and Exhibit 8 presents estimates of the total AII burden for Years 4-6 (1,340 hours) and the total respondent cost ($38,757.28 = total burden hours × the estimated hourly wage for respondents).
Grantee-Level Outcomes Module
The Grantee-Level Outcome Module is required annually of all grantees. Key assumptions related to the burden include:
In Years 4 and 5, we expect that 25 SPF-Rx grantees will complete the Grantee-Level Outcome Data module 1 time each year.
In Year 6, we only anticipate grantees with no-cost extensions will complete the Grantee-Level Outcomes Module. We estimate 7 grantees will request a no-cost extension, and therefore the Year 6 annual burden only includes those grantees.
The Grantee-Level Outcome Module is estimated to take 2.5 hours to complete per response; this includes 1.5 hours to research and compile information, and 1 hour to complete the web instrument.
There are no direct costs to respondents other than their time to complete the instrument.
Six grantees were queried to determine the time it took to complete the Grantee-Level Outcomes Module. The revised burden estimate was calculated based on the experience of those grantees with the full instrument. Subsequently it was determined that several sections of the Grantee-Level Outcomes Module were duplicative of information collected in SPARS. They were removed. This OMB statement specifies new burden estimates based on the reduced instrument. Exhibits 5-7 provide the annual burden detail for the Grantee-Level Outcomes Module for Years 4-6, and Exhibit 8 presents estimates of the Grantee-Level Outcome Module total burden (142.5 hours), and the total respondent cost ($6,215.85 = total burden hours × the estimated hourly wage for respondents).
Community-Level Outcome Data
The Community-Level Outcome Module is required to be reported annually by grantees for all subrecipients. Key assumptions related to the burden include:
In Years 4 and 5, we expect the 25 grantees to report Community-Level Outcomes for 123 subrecipients.
In Year 6, we only anticipate grantees with no-cost extensions will complete the Community-Level Outcomes Module for their subrecipients. We estimate 7 grantees with 32 subrecipients will request a no-cost extension, and therefore the Year 6 annual burden only includes those grantees and subrecipients.
The Community-Level Outcomes Module is estimated to take 1.25 hours to complete per response; this includes .75 hours to research and compile information (to include working with their subrecipients to identify correct data), and .5 hour to complete the web instrument.
There are no direct costs to respondents other than their time to complete the instrument.
Six grantees were queried to determine the time it took to complete the Community-Level Outcomes Module. Revised burden estimate was calculated based on the experience of those grantees with the full instrument. Subsequently it was determined that several sections of the Community-Level Outcomes Module were duplicative of information collected in SPARS. They were removed. This OMB statement specifies new burden estimates based on the reduced instrument. Exhibits 6-8 provide the annual burden detail for the Community-Level Outcomes Module for Years 4-6, and Exhibit 8 presents estimates of the Community-Level Outcome Module total burden 347.5 hours, and the total respondent cost ($15,157.95 = total burden hours × the estimated hourly wage for respondents).
Grantee-Level Interviews
The Grantee-Level Interviews are required in the baseline and final years of the grant funding. In Years 1 and 2 of data collection, 25 SPF-Rx grantees completed the Grantee-Level Interview. The final Grantee-Level Interviews will take place during Year 4 of the evaluation and are estimated to take 1.5 hours to complete, per response. The estimated burden time is based on baseline interviews conducted by the PEPC evaluation team with SPF-Rx grantee staff.
There are no direct costs to respondents other than their time to complete the instrument. Exhibit 6-8 provides the details of the annual burden for the Grantee-Level Interview for Years 4,5, and 6 respectively, and Exhibit 9 presents estimates for years 4, 5, and 6 together of the Grantee-Level Interview total burden (37.5 hours), and the total respondent cost ($1,635.75 = total burden hours × the estimated hourly wage for respondents).
Exhibit 6. Estimates of Annualized Burden for Year 4 of the SPF-Rx Data Collection (N=25 Grantees; N=123 Subrecipients)
Instrument |
Number of Respondents |
Respondent Type |
Responses per Respondent |
Total Number of Responses |
Hours per Response |
Total Burden Hours |
Average Hourly Wage b c |
Total Costd |
Annual Implementation Instrument a
|
123 |
Subrecipient Staff |
1 |
123 |
4 |
492 |
$25.91 |
$12,747.72 |
25 |
Grantee PD or Evaluator |
1 |
25 |
4 |
100 |
$43.62 |
$ 4,362.00 |
|
Grantee-Level Outcomes Module |
25 |
Grantee PD or Evaluator |
1 |
25 |
2.5 |
62.5 |
$43.62 |
$2,726.25 |
Community-Level Outcomes Module |
25 |
Grantee PD or Evaluator |
4.92 |
123 |
1.25 |
153.75 |
$43.62 |
$6,706.58 |
Grantee-Level Interview |
25 |
Grantee PD or Evaluator |
1 |
25 |
1.5 |
37.5 |
$43.62 |
$1,635.75 |
Year 4 Total |
223 |
|
|
321 |
|
845.75 |
|
$28,178.30 |
a The AII is used for both subrecipient and grantee-level reporting. Grantees report AII data at the state or tribal grantee-level, and subrecipients report the AII at the community-level. All other instruments are completed by grantees.
b Grantee PD or Evaluator hourly wage is based on the mean hourly wage for state government managers, as reported in the 2018 Occupational Employment (OES) by the Bureau of Labor Statistics (BLS) found at https://www.bls.gov/oes/current/naics4_999200.htm#11-00000.
c Subrecipient Staff hourly wage is based on the mean hourly wage for local government counselors, social workers, and other community and social service specialists, as reported in the 2018 OES by the BLS found at https://www.bls.gov/oes/current/naics4_999300.htm.
d Total respondent cost is calculated as total burden hours × average hourly wage.
Exhibit 7. Estimates of Annualized Burden for Year 5 of the SPF-Rx Data Collection (N=25 Grantees; 123 Subrecipients)
Instrument |
Number of Respondents |
Respondent Type |
Responses per Respondent |
Total Number of Responses |
Hours per Response |
Total Burden Hours |
Average Hourly Wage b c |
Total Costd |
Annual Implementation Instrument a
|
123 |
Subrecipient Staff |
1 |
123 |
4 |
492 |
$25.91 |
$ 12,747.72 |
25 |
Grantee PDs or Evaluators |
1 |
25 |
4 |
100 |
$43.62 |
$ 4,362.00 |
|
Grantee-Level Outcomes Module |
25 |
Grantee PDs or Evaluators |
1 |
25 |
2.5 |
62.5 |
$43.62 |
$2726.25 |
Community-Level Outcomes Module |
25 |
Grantee PDs or Evaluators |
4.92 |
123 |
1.25 |
153.75 |
$43.62 |
$6706.58 |
Year 5 Total |
198 |
|
|
296 |
|
808.25 |
|
$26,541.83 |
a The AII is used for both subrecipient and grantee-level reporting. Grantees report AII data at the state or tribal grantee-level, and subrecipients report the AII at the community-level. All other instruments are completed by grantees.
b Grantee PD or Evaluator hourly wage is based on the mean hourly wage for state government managers, as reported in the 2018 Occupational Employment (OES) by the Bureau of Labor Statistics (BLS) found at https://www.bls.gov/oes/current/naics4_999200.htm#11-00000.
c Subrecipient Staff hourly wage is based on the mean hourly wage for local government counselors, social workers, and other community and social service specialists, as reported in the 2018 OES by the BLS found at https://www.bls.gov/oes/current/naics4_999300.htm.
d Total respondent cost is calculated as total burden hours × average hourly wage.
Exhibit 8. Estimates of Annualized Burden for Year 6 of the SPF-Rx Data Collection (N=7 Grantees; 32 Subrecipients) a
Instrument |
Number of Respondents |
Respondent Type |
Responses per Respondent |
Total Number of Responses |
Hours per Response |
Total Burden Hours |
Average Hourly Wage c d |
Total Coste |
Annual Implementation Instrument b
|
32 |
Subrecipient Staff |
1 |
32 |
4 |
128 |
$25.91 |
$ 3,316.48 |
7 |
Grantee PDs or Evaluators |
1 |
7 |
4 |
28 |
$43.62 |
$ 1,221.36 |
|
Grantee-Level Outcomes Module |
7 |
Grantee PDs or Evaluators |
1 |
7 |
2.5 |
17.5 |
$43.62 |
$763.35 |
Community-Level Outcomes Module |
7 |
Grantee PDs or Evaluators |
4.57 |
32 |
1.25 |
40 |
$43.62 |
$1,744.8 |
Year 6 Total |
53 |
|
|
78 |
|
213.5 |
|
$7,045.99 |
a The number of grantees and subrecipients included in Year 6 of the data collection represent approximately 25 percent of the SPF-Rx cohort; representing the likely number of grantees and their subrecipients that will request a no-cost extension. All other instruments are completed by grantees.
b The AII is used for both subrecipient and grantee-level reporting. Grantees report AII data at the state or tribal grantee-level, and subrecipients report the AII at the community-level.
c Grantee PD or Evaluator hourly wage is based on the mean hourly wage for state government managers, as reported in the 2018 Occupational Employment (OES) by the Bureau of Labor Statistics (BLS) found at https://www.bls.gov/oes/current/naics4_999200.htm#11-00000.
d Subrecipient Staff hourly wage is based on the mean hourly wage for local government counselors, social workers, and other community and social service specialists, as reported in the 2018 OES by the BLS found at https://www.bls.gov/oes/current/naics4_999300.htm.
e Total respondent cost is calculated as total burden hours × average hourly wage.
Exhibit 9. Total Burden for SPF-Rx Instruments
Instrument |
Respondent Type |
Number of Respondents |
Total Number of Responses |
Total Burden Hours |
Total Cost |
Annual Implementation Instrument a
|
Subrecipient Staff |
123 |
278 |
1,112 |
$ 28,811.92 |
Grantee PDs or Evaluators |
25 |
57 |
228 |
$ 9,945.36 |
|
Total Burden for AII |
|
|
335 |
1,340 |
$38,757.28 |
Grantee-Level Outcomes Module |
Grantee PDs or Evaluators |
25 |
57 |
142.5 |
$6,215.85 |
Community-Level Outcomes Module |
Grantee PDs or Evaluators |
25 |
278 |
347.5 |
$15,157.95 |
Grantee-Level Interview |
Grantee PDs or Evaluators |
25 |
25 |
37.5 |
$1,635.75 |
Total Burden for SPF-Rx |
|
|
695 |
1,867.5 |
$61,766.83 |
a The AII is used for both subrecipient and grantee-level reporting. Grantees report AII data at the state or tribal grantee-level, and subrecipients report the AII at the community-level.
There are no respondent costs for capital or start-up or for operation or maintenance.
SAMHSA plans to allocate resources for the management, processing, and use of the collected information in a manner that will enhance its utility to agencies. The contract award to cover this evaluation is $1,619,322 over a 12-month period. Thus, the annualized contract cost is $1,619,322. It is estimated that two SAMHSA employees will each be involved for 15 percent of their time, at an estimated annualized cost of $32,670 to the government. The total estimated average cost to the government per year is $1,651,992.
Currently, there are 618 total burden hours. SAMHSA is requesting 1,867.5 burden hours. This is an adjustment of 1,249.5 hours based on actual burden estimates revisions to data collection instruments. This equates to an overall increase of 1,249.5 hours and $15,504.06 over the previously approved data collection. The annual burden costs have increased from previous approval due to 4 factors:
The previous OMB approval estimated 100 AII responses (25 grantees and 75 subrecipients), but the SPF-Rx grantees currently have 25 grantees and 123 subrecipients, totaling 148 AII responses, accounting for an additional 48 responses, 192 hours and $4,974.72 per year of data collection with the entire grantee cohort (i.e., Years 4 and 5).
The previous OMB approval estimated only 25 Community-Level Outcomes Module responses, but one is completed for each subrecipient, and therefore the burden has been increased to 123 responses, accounting for an additional 98 responses, 343 hours, and $14,961.66 per year of data collection with the entire grantee cohort (i.e., Years 4 and 5).
Based on grantee feedback, the time it takes for respondents to complete the AII was increased from 2.3 hours to 4 hours. While grantee feedback indicated that the Grantee-Level Outcomes Module burden should be increased from 3 hours to 6.2 hours, the module was reduced and the burden decreased to 2.5 hours. The Community-Level Outcomes Module burden was initially increased from 3 hours to 3.5 hours based on grantee feedback, but the instrument was subsequently reduced and the burden now estimated at 1.25 hours. The adjustments in the three instruments results in an overall reduction of 55 hours and $2,248.4 per year of data collection with the entire grantee cohort (i.e., Years 4 and 5). This reflects a 42% decrease in burden for the Grantee-Level Outcomes and the Community-Level Outcomes Modules (decrease from 6 hours to 3.75 hours across all respondents in all years). We expect that these burden estimates are more accurate than previous burden estimates, given that they are based on feedback from those who have completed data collection efforts during Years 1 and 2 of the evaluation in addition to changes in the data collection instruments based on the identification of duplicative data elements. Please note, this estimate only accounts for the changes in instrument burden totals for the number of respondents accounted for by the previous data collection. It does not include additional burden hours and costs in factors 1 and 2, above, that were added due to changes in the number of respondents. The additional burden hours and costs in factors 1 and 2 were calculated using these new per response burden estimates.
The SPF-Rx hourly wage estimates in the previous statement were $40.88, based on one salary using 2015 Bureau of Labor Statistics (BLS) data. The current package increased grantee-level staff to 2018 BLS data for hourly wages of $43.62. BLS data are an accurate representation of salary data, utilizing the National Compensation Survey which provides comprehensive measures of compensation cost trends and is updated each year to reflect those changes in average compensation. Using the higher state-level average hourly wage (not the BLS data for local government for subrecipient staff), this wage increase would account for an additional $2,307.08 to the previous data collection (calculated using total burden hours X difference in hourly wage). Additionally, for subrecipient staff we used the 2018 BLS data for local government, which is $25.91, a more realistic wage for subrecipient staff. If the previous data collection had used the lower local government rate for the estimated 75 subrecipients submitting the AII, the total burden would decrease by $4,491.00 (calculated using total burden hours for subrecipient AIIs X difference in the hourly wage). This estimate only accounts for the changes in hourly wages for the number of respondents accounted for by the previous data collection. It does not include additional burden hours and costs in factors 1 and 2, above, that were added due to changes in number of respondents.
Exhibit 10 represents a timeline for data collection and reporting benchmarks for the SPF-Rx Evaluation.
Exhibit 10. Time Schedule for SPF-Rx Evaluation Data Collection
Activity |
Time Schedule |
Year 4 AII, Grantee- and Community-Level Outcomes Data Collection |
November 2020–January 2021 |
Grantee Level Interview |
February-March 2021 |
Disseminate Findings: Year 5 Annual Report |
July 2021 |
Year 5 AII, Grantee- and Community-Level Outcomes Data Collection |
November 2021-January 2022 |
Disseminate Findings: Year 5 Annual Report |
July 2022 |
Year 6 AII, Grantee- and Community-Level Outcomes Data Collection (For grantees with no-cost extensions) |
November 2022-January 2023 |
Disseminate Findings: Year 6 Annual Report |
July 2023 |
The PEPC evaluation team will use the data collected through the SPF-Rx evaluation to help SAMHSA reach its diverse stakeholders. The objective for all reports and dissemination products is to provide user-friendly documents and presentations that help SAMHSA successfully disseminate and explain the findings. The dissemination plan includes products in a variety of formats for a variety of target audiences, such as:
Annual reports that summarize findings. The SPF-Rx reports will include brief profiles on each grantee, with helpful and easy-to-read graphics on performance data, rather than lengthy text.
Briefings for SAMHSA and other federal stakeholders Audiences for briefings may include SAMHSA staff, grantees, and other stakeholders.
Aggregate information may also be used in journal articles, scholarly presentations, budget justifications, and other testimony related to the outcomes of the SPF-Rx program.
SPF-Rx Evaluation
The PEPC SPF-Rx evaluation uses a series of interdependent analyses to answer the key EQs developed to assess the impact of the SPF-Rx program on prescription drug misuse, opioid overdoses, and related outcomes. The evaluation will fully incorporate all data from the cross-site evaluation instruments, as well as secondary data, including CDC Wide-ranging Online Data for Epidemiologic Research (CDC WONDER) overdose mortality data, NSDUH consumption data, SPARS progress report data and strategic plans, and National Poisoning Data System poisoning data. The analysis plan includes a range of approaches from basic descriptive analyses of GPRA measures, grantee performance measures, and National Outcomes Measures (NOMs; e.g., means, frequencies, percentages, trend analysis), to sophisticated qualitative analysis and multiple quantitative analytic frameworks and models that reflect complexities that are anticipated to arise with data collected by the PEPC team.
Matched Comparison Groups: The SPF-Rx evaluation will use a pre- and post-design, with matched comparison groups when relevant and feasible. The PEPC team plans to obtain key county-level characteristics from baseline census, archival, and survey data sources and use that information to select comparison counties (or communities). For all grantees, the required estimates will be available through standard public reporting. Under no circumstances will new data collection be required for the matching process. Follow-up outcomes data for the matched comparison groups will come from the same data sources used for the matching process.
Quantitative Analyses: Several features of the evaluation design and EQs guided the selection of the analysis frameworks that the SPF-Rx evaluation has used or adapted the following features:
repeated outcomes;
data from state and tribal grantees;
data from communities nested within grantees;
nonrandomized comparison of communities within grantee states; and
nonrandom selection of intervention types that often occur in combination.
Outcome Evaluation Models: For this exercise, a difference in differences (DiD) analysis will be undertaken, in addition to cross-site analytic approaches, as appropriate. In the DiD approach, the relative gain or loss on the selected outcome measures available for paired communities over the same time frame are tracked, and the average change in the comparison community is subtracted from the average change in the SPF-Rx community. When executed properly, this approach isolates the effects of the intervention from historical trends among the treatment and control communities. For example, for an analysis of outcomes of an intervention we will use the poison control data (available at the zip code level) for the intervention area and an adjacent area not implementing the intervention activity.
When estimating intervention effects using a DiD approach, the data are assembled over time in a panel of data points; this trend data must be “wide” enough that the researcher has sufficient information to estimate an effect. In the example above, poison control data are available historically and provide a long enough panel of data points for the analysis. By including community-level fixed effects, the researcher can also mechanically remove variation across the comparison communities from the estimation, which can interfere with a valid estimation of the effect. If the estimated effects appear to be small or imprecise, the team will consider random effects models to estimate the intervention using explanatory variation from across and within communities. If properly specified, the random effects model is more precise than the fixed effects model. The drawback to a random effects model is that it is more vulnerable to confounding influences like state-level policies and the team will include covariates to adjust for possible confounders in our analysis.
OMB approval expiration dates will be displayed.
There are no exceptions to the certification statement. The certifications are included in this submission.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | author |
| File Modified | 0000-00-00 |
| File Created | 2021-01-14 |