Ss_a_pfs_mrt 12.22.15

PARTNERSHIPS FOR SUCCESS PROGRAM EVALUATION FOR PREVENTION CONTRACT

SUPPORTING STATEMENT

1. JUSTIFICATION

   1. Circumstances of Information Collection

The Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Substance Abuse Prevention (CSAP) is requesting approval from the Office of Management and Budget (OMB) for data collection activities related to the performance monitoring of SAMHSA’s Strategic Prevention Framework (SPF) Partnerships for Success (PFS) program. The SPF-PFS program aims to address two of SAMHSA’s top substance abuse prevention priorities: underage drinking (UAD; age 12 to 20) and prescription drug misuse and abuse (PDM; age 12 to 25). While used primarily for performance monitoring purposes, data collected through the tools described in this statement will also be used by SAMHSA’s Program Evaluation for Prevention Contract (PEP-C) national cross-site evaluation of the SPF-PFS programs. PEP-C is scheduled through September 2018 to systematically collect and maintain community subrecipient information, quarterly progress reports and outcomes data submitted by SPF-PFS grantees through the online PEP-C Management Reporting Tool (MRT) (Attachment 1). SAMHSA is requesting approval for data collection through the PEP-C MRT using the instruments listed below:

1. Contact Information: this instrument includes sections for Grantee Information, Grantee Staff, Sub-State Information, Community Subrecipient information, and Subrecipient Staff

2. Quarterly Progress Report (QPR): This instrument will gather data related to implementation of the SPF-PFS grant based on the SPF steps (Assessment, Capacity, Planning, Implementation, and Evaluation).

3. Outcome Data: this instrument includes 4 separate sub-instruments that grantees will complete in varying time frames dependent on requirements.

   1. Grantee Target Outcome Data

   2. PFS Selected Grantee-Level Outcome Data

   3. Community-Level Outcome Data for Subrecipients

   4. Substitute Data Source Request

SAMHSA’s SPF-PFS program is authorized under Section 516 of the Public Health Service Act, as amended, and addresses the Healthy People 2020 Objective of Substance Abuse. SPF-PFS also supports two of SAMHSA’s eight Strategic Initiatives: (1) the Prevention of Substance Abuse and Mental Illness and (2) Data, Outcomes, and Quality. Finally, this effort supports and is aligned with the Surgeon General’s “Call to Action to Prevent and Reduce Underage Drinking 2007,” the Department of Health and Human Services’ National Prevention Strategy 2011, and the Office of National Drug Policy’s (ONDCP) “Epidemic: Responding to America’s Prescription Drug Abuse Crisis 2011.”

Background:

Over the past decade, a large number of evaluation studies demonstrated that prevention interventions effectively reduce substance abuse, as well as delinquent behaviors; violence; and other mental, emotional, and behavioral health problems (e.g., Calear & Christensen, 2010; Lemstra et al., 2010; Ttofi & Farrington, 2011). Among 12- to 20-year-olds from 2002 to 2012, rates of current alcohol use decreased from 28.8% to 24.3%, rates of binge drinking declined from 19.3% to 15.3%, and heavy alcohol use declined from 6.2% to 4.3% (SAMHSA, 2013b). Despite these successes, UAD continues to be a significant public health problem. The 2012 National Survey on Drug Use and Health (NSDUH) report estimates that 9.3 million persons aged 12 to 20 drank alcohol in the past month, 5.9 million binge drank, and 1.7 million drank alcohol heavily (SAMHSA, 2013b). UAD causes serious harm to the adolescent drinker as well as to the community as a whole (Office of Juvenile Justice and Delinquency Prevention, 2012). Alcohol use by adolescents negatively effects brain development, results in other serious health consequences (e.g., alcohol poisoning, risky sexual behaviors, and addiction), and leads to safety consequences from driving under the influence, poisonings, and other injuries. UAD places youth at increased risk for violence perpetration and victimization along with social or emotional consequences (e.g., low self-esteem, depression, anxiety, lack of self-control, stigmatization by peers), academic consequences (e.g., poor academic performance, truancy, suspension or expulsion from school), and family consequences (e.g., poor relationships with parents).

Adolescent drinking can also impose economic consequences, ranging from personal costs (e.g., payment for alcohol treatment or medical services) to familial costs (e.g., parents taking time off of work to drive children to treatment) to community costs (e.g., providing enforcement, supervision, or treatment to underage drinkers). In 2001, the sale of alcohol to underage drinkers led to an estimated 3,170 deaths and almost 2.6 million injuries, which caused an estimated $61.9 billion in harm to society through medical spending, property losses, lost wages, and the loss of quality-adjusted life years (Miller, Levy, Spicer, & Taylor, 2006). Sacks et al. (2013) estimated that in 2006, UAD was responsible for $24.6 billion (11%) of the total cost to society of excessive alcohol consumption in the United States.

PDM among young people is the fastest-growing drug problem in the United States, and prescription drugs are second only to marijuana as the drugs most abused by teens (National Institutes of Health, 2011). PDM refers to the use of licit drugs to treat pain, attention deficit disorder, or anxiety without a prescription; in a way other than prescribed; or because of the feelings it may elicit (National Institutes of Health, 2011). Recent NSDUH estimates indicate that approximately 5% of respondents age 18 to 25 and about 2.5% of respondents age 12 to 17 report PDM (SAMHSA 2012a, 2013b). The widespread consequences of PDM are consistent with the consequences caused by UAD (i.e., violence perpetration and victimization along with health, safety, social, emotional, academic, familial, and economic consequences). In 2011, youth age 12 to 24 accounted for approximately 23% of emergency department visits involving nonmedical use of pharmaceuticals (SAMHSA, 2013a). Since 2003, more deaths have been due to opioid analgesic overdoses than heroin and cocaine combined (Centers for Disease Control and Prevention, 2012).

Strategic Prevention Framework Partnerships for Success Program

In 2004, SAMHSA began funding the SPF-SIG, an infrastructure grant program designed to help States, jurisdictions/territories, and tribal organizations implement the SPF, with the goals of preventing the onset and reducing the progression of substance abuse, reducing problems related to substance abuse, and building capacity and infrastructure for prevention. Grantees used the SPF model, which consists of five steps: (1) needs assessment; (2) capacity building; (3) strategic planning; (4) implementation of programs, policies, and practices; and (5) evaluation. Grantees also considered cultural competence and sustainability at each step in the process. SAMHSA awarded 5-year SPF-SIG grants to 49 States, 19 tribal organizations, 8 jurisdictions/territories, and the District of Columbia from 2004 to 2010. These grantees funded approximately 650 subrecipient communities within their regions to also adopt the SPF process, build their prevention capacity, and implement substance use prevention interventions.

In 2009 SAMHSA initiated the SPF-PFS program, the focus of this data collection, to provide an opportunity for SPF-SIG recipients to expand on their SPF efforts in their communities by promoting further alignment of resources and priorities and improvement of prevention infrastructure. The first SPF-PFS cohort, PFS I, included five grantees, each funded for 5 years to target one statewide priority need that would lead to a lasting reduction in substance use (the PFS I cohort is not a part of the current data collection). PFS II (n=15), PFS 2013 (n=16), PFS 2014 (n=21), and PFS 2015 (n=32) cohort grantees must select one or both of two national prevention priorities identified by SAMHSA: (1) UAD among persons age 12 to 20 and (2) PDM among persons age 12 to 25.

The overall goals of for the PFS II, PFS 2013, PFS 2014, and PFS 2015 cohorts are to:

• Prevent the onset and reduce the progression of substance abuse, prioritizing UAD among persons age 12 to 20, PDM among persons age 12 to 25, or both;

• Reduce substance abuse-related problems;

• Strengthen prevention capacity and infrastructure at the State and community levels; and

• Leverage, redirect, and align statewide funding streams and resources for prevention.

PFS II, PFS 2013, PFS 2014, and PFS 2015 cohort grants also place a new emphasis on community need in that grant awardees (States, jurisdictions/territories, and/or tribal organizations) must identify and fund high-need and low-capacity subrecipient communities to implement evidence-based prevention programs, policies, and practices that address the selected prevention priorities. Please see Exhibit 1 for the number of Grantees in each relevant PFS cohort as well as the actual or expected number of community subrecipients funded by the grantees in each cohort.

Exhibit 1. Cohorts of the Strategic Prevention Framework Partnerships for Success (SPF-PFS) Who Will Enter Data in the Program Evaluation for Prevention Contract (PEP-C) Management Reporting Tool (MRT)

Cohort	# of Grantees	# of Subrecipient Communities	Length of Grant	Start Date–End Date
PFS II	15*	~140	3 years	Oct. 2012–Sept. 2015
PFS 2013	16**	~240	5 years	Oct. 2013–Sept. 2018
PFS 2014	21***	~250	5 years	Oct. 2014—Sept. 2019
PFS 2015	32****	~338	5 years	Oct. 2015—Sept. 2020
Total	69****	~828

* Includes 14 States and 1 territory.

**Includes 14 States and 2 territories.

*** Includes 12 States, 3 territories, 5 tribal organizations, and the District of Columbia

**** Includes 21 States, 3 territories, and 8 tribal organizations; all 15 PFS II grantees received funding as PFS 2015 grantees so this total counts those grantees and subrecipients only once.

Potential Impact of SPF-PFS Performance and Outcome Measure Findings

The instruments covered in this package were based are modified versions of similar instruments used for performance and outcomes monitoring of earlier programs such as SPF SIG. Two national SPF SIG cross-site evaluations were conducted: one for grantee cohorts I and II and the other for grantee cohorts III, IV, and V. Published results from the Cohorts I and II evaluation focus mainly on changes to prevention infrastructure over the course of SPF-SIG funding. Orwin and colleagues (2014) found that, on average, grantees’ strategic planning, workforce development, and support of the implementation of evidence-based program, policies, and practices increased during their grant period. Grantees showed continued improvement in these areas, as well as in evaluation and monitoring, one year after their SPF-SIG funding ended. Cohort I and II grantees also showed improvements in integration (cooperation across state agencies and across state, regional, and local levels) over the course of their grants (Orwin et al., 2014). Outcome data for Cohort III grantees demonstrated mixed findings: 30% of grantees and 32% of subrecipients showed improvement in past 30-day alcohol use, and 50% of grantees and 29% of subrecipients showed improvements in substance-related consequences (e.g., alcohol-involved motor vehicle fatalities, drug-related arrests; SAMHSA, 2012b). Currently, many grantees in SPF-SIG Cohorts IV and V are in the process of completing or just completed their grants so available findings related to those cohorts provide only baseline and basic descriptive information.

Thus far an examination of the outcomes of the SPF-SIG cohorts has been limited. In addition to serving a monitoring purpose, the outcomes monitoring measures described in this statement will allow more extensive outcomes evaluation of the PFS grantees. For SPF-SIG, analyses showed that implementing a greater proportion of population-based versus individual-based interventions is related to a community having other prevention funding sources and having law enforcement more involved in the planning process. While the already OMB approved PFS cross-site measures will provide more detailed information about the intervention types, the outcomes monitoring measures described in this statement will allow the PFS cross-site to move beyond looking at process measures and assess the linkage between the intervention type and outcomes. Similarly, the SPF-SIG cross site evaluation provided some descriptive analyses of funding, but the PFS QPR and outcomes measures will allow SAMHSA to tie leveraging of funding to outcomes.

This data collection will place a new emphasis on the SPF-PFS impact on outcomes related to PDM, including the prevalence of prescription drug misuse and related consequences such as prescription drug poisonings and overdoses. The recent emergence of PDM as a serious public health issue also provides a unique opportunity for the SPF-PFS cross site evaluation to examine the implementation and effectiveness of prevention interventions developed to target this issue. In addition, this performance monitoring data collection and the SPF-PFS cross-site will more strongly emphasize an examination of economic issues, including associations between funding and outcomes. Through the capacity section of the QPR described in this statement, grantees will provide information on leveraged funding, or all of the sources of funding that support both their overall substance abuse prevention efforts and their PFS-specific efforts. These sources of support can include other federal grants, as well as state, local, foundation, and corporate sources, along with donations.

The SPF-PFS performance monitoring and outcomes measurement and related cross site evaluation are expected to have numerous program and policy implications and outcomes at the national, state, and community levels. They will provide valuable information to the prevention field about best practices in real world settings, along with providing guidance to governmental entities and communities as to what types of interventions should be funded and implemented to reduce UAD and PDM. Information and guidance about leveraging that comes from the SPF-PFS cross site evaluation will allow the federal government, state, tribes, jurisdictions, and local communities to more effectively and efficiently use their resources and sustain future prevention efforts.

2. Purpose and Use of Information

The SPF-PFS performance monitoring measures will primarily be tools for SAMHSA project officers to systematically collect data to monitor grant program performance and outcomes along with grantee technical assistance needs. In addition to assessing activities related to and progress through the SPF steps, the performance monitoring instruments covered in this statement collect data to assess the following grantee required specific performance measures:

• Number of training and technical assistance activities per funded community provided by the grantee to support communities;

• Reach of training and technical assistance activities (numbers served) provided by the grantee;

• Percentage of subrecipient communities that submit data to the grantee data system

The instruments also collect data to provide information for the following PFS required Government Performance and Results Act (GPRA) measure:

• Number of sub-recipient communities that improved on one or more targeted NOMs indicators (Outcome)

To take advantage of lessons learned through performance monitoring data collection conducted for the SPF SIG programs, PFS performance monitoring instrument development began with the related SPF SIG and PFS I instruments. The PFS performance monitoring instrument development process also included input from grantee-level SPF PFS evaluators, SAMHSA management and PFS Project Officers, and other stakeholders (see Exhibit 11 – the statistical consultants list). After careful review, revisions were made to streamline the instruments, decrease verbosity, create consistency in assessing infrastructure at the grantee and community subrecipient levels, and address gaps such as those related to leveraged funding. In order to reduce burden, information collected through the instruments covered by this statement will also provide data to the PEP-C national cross-site evaluation of the PFS program. The overall goal of the cross-site evaluation is to document and assess the factors that contribute to the effectiveness of the PFS approach to SAMHSA’s mission of reducing UAD and PDM, including costs, inputs, outputs, and contextual factors.

SPF-PFS Management Reporting Tool (MRT; Attachment 1)

The SPF-PFS MRT primarily will focus on allowing SAMHSA PFS Project Officers to monitor grantees to assure they follow the SPF process through the specific instruments described next. Unless otherwise noted grantees provide all of the described information. The SPF-PFS MRT gathers all information through a web-based data collection system that uses clickable radio buttons, check boxes, drop-down choice items, and open-ended text boxes, as relevant. It also allows grantees to upload required documents requested by their Project Officers.

After accessing the system, the MRT will direct users to the Home page. The Home page includes the following standard functions found on each page throughout the system:

1. The home navigation page will provide access to the 4 landing pages: Contact Information page (1), Quarterly Progress Reporting page (2), Outcome Data page (3), and Cross-Site Instruments page (4; the cross-site instruments are covered by OMB No. 0930-0348). The top navigation menu provides drop-downs which can be used to access pages directly from any other page.

2. The linked “bread crumb trail” will identify where users are in the system and provide the ability to move backwards to previous sections.

Contact Information: This instrument includes sections for Grantee Information (provided by SAMHSA), Grantee Staff (these staff will have access to the system for data entry purposes; information collected includes roles, name, e-mails, and telephone numbers), Sub-State Information (these are intermediaries between grantees and their community subrecipients; information collected includes name, funding amount, and target areas/geographic locations), Community Subrecipient information (information collected includes type, name, location, target area, and funding status/information), and Subrecipient Staff (information collected includes roles, name, e-mails, and telephone numbers; these staff will have access to the system to respond to cross-site data collection instruments).

Quarterly Progress Report (QPR): This web-based data collection instrument is usually completed by the grantee Project Director once each quarter (or four times each year). The instrument will gather data related to implementation of the SPF-PFS grant based on the SPF steps (Assessment, Capacity, Planning, Implementation, and Evaluation) along with how Health Disparities are addressed through each step. Clicking the link for each step or section will direct the user to the relevant landing page. For example, the “Assessment” link will direct user to the Assessment landing page (Page ID 2.1).

Data collected will include information on accomplishments and barriers for each step. The Capacity section (Page ID 2.2) also collects information on workgroup membership and meetings to assess leveraging of partnerships; grantee-level funding and in kind resources to assess leveraging of funds from various sources; and training received by grantees and provided to subrecipients by grantees including training topics, numbers reached, delivery sources, and unfulfilled training needs. The Planning section (Page ID 2.3) also allows grantees to upload their PFS strategic plans as those become available. The Implementation section (Page ID 2.4) also requests grantees to provide information on the progress of each of their community subrecipients. The Evaluation section (Page ID 2.5) also allows grantees to upload their PFS evaluation plans or PFS local evaluation reports as those become available. The Health Disparities section (Page ID 2.6) allows SAMHSA project officers to monitor grantee efforts to fulfill requirements related to the Affordable Care Act that grantees address health disparities related to substance abuse risks, prevalence, and outcomes. This section allows grantees to upload required health disparities impact statements (plans for how they will address health disparities through PFS efforts) as well as describe health disparities-related activities, accomplishments and barriers relevant to each one of the SPF steps.

Clicking the “Quarterly Progress Report Submission” link will direct user to the Quarterly Progress Report Submission listing page (Page ID 2.7).

Outcome Data: This instrument includes 4 separate sub-instruments that grantees will complete at varying time points dependent on requirements.

Grantee Target Outcome Data: On this instrument grantees indicate their target outcomes from a check list of substance use related consumption, consequence, and intervening variables (see Page ID 3.1). Generally grantees will only enter data on this instrument once during the course of their grant. Through this instrument grantees can also download Excel files containing state-level longitudinal outcomes data consolidated by PEP-C from national databases.

PFS Selected Grantee-Level Outcome Data: This instrument allows grantees to provide supplemental state-level outcomes data to PEP-C (see Page ID 3.2). These data are intended to reflect changes at the grantee level in the consumption, consequence, and intervening variables targeted by the PFS grant program. It is not required and most often used by Pacific jurisdiction or tribal organization grantees who are not included in the national database data consolidated by PEP-C and provided through Grantee Target Outcome Data page. Some state grantees may also choose to enter data on this page, if for example they want to report supplemental state-level data from a state-specific survey. Pacific jurisdiction or tribal grantees who consider themselves single-community grantees (i.e. they may fund sub-contractors to implement interventions but do not fund separate community subrecipient organizations) enter their outcomes data in the Community-Level Outcome Data for Subrecipients page rather than on the PFS Selected Grantee-Level Outcome Data page.

The outcomes provided on this instrument come from existing survey and administrative data within the state, tribe, or jurisdiction. The instrument requests descriptions of the outcome measures (target substance and outcome; data source type and name; reported outcome calculation description; item and response wordings; sample/population age and grade parameters; time frame of data collection; and actual outcome values and variability estimates).

Community-Level Outcome Data for Subrecipients: Grantees will use this instrument to provide required data about consumption, consequence, and intervening variable outcomes of their community subrecipients’ SPF-PFS activities. These outcomes requirements are outlined in Exhibit 2. Grantees will provide one annual outcome for each of their community subrecipients for each of the following: 1) underage drinking; 2) prescription drug misuse; and 3) any other targeted substance. In addition they must provide at least two time points of data (pre-PFS and towards the end of their grant) for two additional underage drinking measures and two additional prescription drug misuse measures.

Exhibit 2. Community-Level Outcomes Requirements for SPF-PFS Grantees

Note: Grantees targeting additional substances (e.g. marijuana) must provide 1 annual exact or approved substitute measure for each additional target.

The outcomes provided on the Community-Level Outcome Data for Subrecipients instrument come from existing survey and administrative data within the state, tribe, or jurisdiction. The instrument requests descriptions of the outcome measures (target substance and outcome; data source type and name; reported outcome calculation description; item and response wordings; sample/population age and grade parameters; time frame of data collection; and actual outcome values and variability estimates). Grantees complete a separate instrument for each subrecipient and each outcome.

When they are ready to begin entering outcome data on their community subrecipients grantees first select one of their subrecipient from a dropdown menu (see Page ID 3.3). For a new outcome, grantees click on the Add a Record button. Once they have added records, they will be able to view previously added records for the selected Subrecipient. The PEP-C MRT reduces burden on this instrument through two processes:

• By allowing grantees to copy information on an outcome from one subrecipient to another. Then grantees just need to provide the subrecipient-specific information on the outcome values and variability.

• After the initial data entry for a subrecipient, grantees only need to provide information on the follow-up data points time frame along with the value calculation information at that time point. They just click on the Add Follow-Up Data link provided on the page (see Page ID 3.3)

Substitute Data Source Request: This instrument allows grantees to request permission from SAMHSA to use “substitute measures” for their community outcomes – i.e. measures that differ from a list of pre-approved outcomes measures (see Page ID 3.4). Grantees can consult the PEP-C PFS Community Outcomes Manual, tables 3 and 4 for definitions of the required outcomes (see https://pep-c.rti.org/HERO/KB/PEP-C-KB/Content/PFS/Community%20Outcomes/CO-Guidance-Manual/Community-Outcomes-Guidance-Manual.pdf)

Grantees are only required to submit a substitute data request for measures they are using to meet the annual exact measure requirement (one annual measure for alcohol, one for prescription drugs, and one for any additional target substance for each community). They do not need to submit a substitute data request for other measures. In practice this means that about 10 grantees submit a substitute data source request each year.

The instrument requests descriptions of the proposed substitute outcome measures (target substance and outcome; data source name; exact wording of proxy item, response options, and reported outcome; formula for calculating the outcome; reasons for the substitute data source request; data collection information such as time frame, sample size, sampling strategy, response rates, and method; and reliability and validity information).

SPF-PFS MRT and the PFS National Cross-Site Evaluation

As stated previously, in order to reduce burden, information collected through the instruments covered by this statement will also provide data to the PEP-C national cross-site evaluation of the PFS program. Exhibit 3 outlines where the SPF-PFS MRT items fit in with the PFS national cross-site evaluation questions and constructs. Constructs and descriptions that are bolded represent specific information needed for GPRA or performance measures.

Exhibit 3. Program Evaluation for Prevention Contract: Partnerships for Success Management Reporting Tool Sections and Items by Cross-site Evaluation Questions and Constructs

EQ1. Was the implementation of PFS programs associated with a reduction in underage drinking and/or prescription drug misuse and abuse?
Construct	Data Source	MRT Page ID Location and Description
Grantee-level outcomes: Intervening variables (i.e., perception of parental or peer disapproval, perceived risk or harm of use, family communication); Substance use (i.e., 30-day alcohol use, prescription drug misuse, and binge drinking); Consequences (i.e., alcohol and/or drug-related car crashes and injuries; alcohol- and drug-related crime; alcohol and prescription drug related ER visits; overdose/poisoning)	Secondary data from the National Survey of Drug Use and Health (NSDUH), Fatality Analysis Reporting System (FARS), National Public Education Financial Survey, and Uniform Crime Reporting (UCR) Program; provided by PEP-C in the MRT OR provided by the grantee	MRT 3.1-3.2.1 (data source, targeted substance, targeted outcome, data source type, reported outcome, response options, population/sample parameters, data time frame, number of events, denominator, calculated value, value type, variability)
Subrecipient-level (community) outcomes: Intervening variables (i.e., perception of parental or peer disapproval, perceived risk or harm of use, family communication); Substance use (i.e., 30-day alcohol use, prescription drug misuse, and binge drinking); Consequences (i.e., alcohol and/or drug-related car crashes and injuries; alcohol- and drug-related crime; alcohol and prescription drug related ER visits; overdose/poisoning)	Secondary data from administrative and survey data sources reported by grantees in the MRT in the Community-Level Outcomes Data for Subrecipients section	MRT 3.3-3.3.1 (data source, targeted substance, targeted outcome, data source type, reported outcome, response options, population/sample parameters, data time frame, number of events, denominator, calculated value, value type, variability) MRT 3.4-3.4.3 (substitute data source request for measures that do not meet exact annual outcomes requirements – reason for request, data source, measure description) MRT 3.5 (substitute data source request approval – completed by SAMHSA project officers)
EQ2: Did variability in the total level of funding from all sources relate to outcomes? Did variability in the total level of PFS funding relate to outcomes, above and beyond other funding available to communities?
Construct	Data Source	MRT Page ID Location and Description
Grantee-level funding	Grantee quarterly progress reports (QPR)	MRT 2.2.4.-2.2.4.1 (funding; leveraged funding - other funding by source & in-kind labor and other resources)
Subrecipient-level funding	Grantee QPR	MRT 1.3-1.3.1 (sub-state funding) MRT 1.4.1 (subrecipient date funded; amount of award per year; other funding streams)
EQ3. What intervention type, combinations of interventions, and dosages of interventions were related to outcomes at the grantee level? What intervention type, combinations of interventions, and dosages of interventions were related to outcomes at the community level?
NO RELEVANT ITEMS IN THE MRT (aside from outcomes)
EQ4. Were some types and combinations of interventions within communities more cost effective than others?
NO RELEVANT ITEMS IN THE MRT (aside from outcomes)
EQ5. How does variability in factors (infrastructure, subrecipient selection, intervention selection, implementation, geography, demography, training/technical assistance [T/TA], barriers to implementation) relate to outcomes across funded communities?
Construct	Data Source	MRT Page ID Location and Description
Infrastructure	Grantee QPR	MRT 1.4.1 (subrecipient type & past SPF-SIG recipient) Page ID 1.4.2 (subrecipient submits data to grantee data system)
	Grantee QPR	MRT 2.2.1-2.2.3 (council, workgroup, and sub-committee members, organizations, sectors represented, meeting types & topics, and subcommittees)
Subrecipient selection	Grantee QPR	MRT 1.4-1.4.1 (subrecipient information incl. type, # of subrecipients)

Note. NOMs, National Outcomes Measures; MRT, Management Reporting Tool; PFS, Partnerships for Success; SPF, Strategic Prevention Framework; TA, Technical Assistance.

Exhibit 3. Program Evaluation for Prevention Contract: Partnerships for Success Management Reporting Tool Sections and Items by Cross-site Evaluation Question and Constructs (cont.)

EQ5. How does variability in factors (infrastructure, subrecipient selection, strategy selection, implementation, geography, demography, training/technical assistance [T/TA], barriers to implementation) relate to outcomes across funded communities? (cont.)
Construct	Data Source	MRT Page ID Location and Description
Training and technical assistance (TA)	Grantee QPR	MRT 2.2.5-2.2.5.1 (training/TA received by grantees – topic, source, delivery mechanism, timeliness, effectiveness, description, date closed)
		MRT 2.2.5-2.2.5.1 (training/TA provided to subrecipients – topic, source, # individuals and subrecipients served, delivery mechanism, timeliness, effectiveness, description, date closed)
Barriers to implementation	Grantee QPR	MRT 2.1-2.1.1 (assessment barriers) MRT 2.2.6-2.2.6.1 (capacity barriers) MRT 2.3.2-2.3.2.1 (planning barriers) MRT 2.4.1-2.4.1.1 (implementation barriers) MRT 2.5.4-2.5.4.1 (evaluation barriers)
Additional Monitoring Measures
Construct	Data Source	MRT Location and Description
Progress through SPF steps	Grantee QPR	MRT 2.1-2.1.1 (assessment accomplishments) MRT 2.2.6-2.2.6.1 (capacity accomplishments) MRT 2.3-2.3.2.1 (strategic plan upload and planning accomplishments) MRT 2.4.1-2.4.1.1 (implementation accomplishments) MRT 2.4.2 (community subrecipient progress) MRT 2.5.1-2.5.4.1 (evaluation plan and report uploads; evaluation accomplishments)
Health disparities	Grantee QPR	2.6-2.6.2.1 (grantee accomplishments and barriers; health disparities document upload)

Note. NOMs, National Outcomes Measures; MRT, Management Reporting Tool; PFS, Partnerships for Success; SPF, Strategic Prevention Framework; TA, Technical Assistance.

3. Use of Information Technology

Grantee staff will provide information on the SPF-PFS Management Reporting Tool through the PEP-C online data collection system. Using a Web instrument allows for automated data checks as well as for skip procedures and prepopulated fields based on prior responses to certain questions. This will reduce the burden among respondents and data entry error, thereby increasing the efficiency of data entry and improving data quality. The automated data checks will ensure that responses follow the expected format (e.g. numbers or dates where those are expected). The Web based system allows grantees to copy information from one form to another and then change information as needed, such as when grantees need to provide similar community outcomes data on the same measures for multiple subrecipient communities, where only the outcomes value differs. Similarly, once completed initially, some items are automatically pre-populated, such as when Grantees provide measure description information on baseline community outcomes data and then only need to change the time frame and outcomes values at later time points.

The Web-based system also allows SAMHSA PFS Project Officers to review submissions conveniently, request revisions as needed, and then provide approvals to grantees on their submissions as relevant.

A dashboard and other reports will also be available to SAMHSA and the PEP-C team, as well as the grantees and subrecipients who submit data, so that they can monitor the overall status of data collection and monitor performance. Grantees will have access to their own data.

The Web-based system also allows grantees and SAMHSA Project Officers easy access to the PEP-C Knowledgebase, which contains data submission manuals and other relevant documents, a section with responses to frequently asked questions, and a link to a Technical Assistance Submission form (see https://pep-c.rti.org/Default.aspx?tabid=183). Grantees and Project Officers can also request technical assistance on their SPF-PFS data entry through e-mail and a phone request system. All technical assistance requests are routed to one electronic system which keeps track of requests, follow-ups, and resolutions.

4. Effort to Identify Duplication

This monitoring tool is collecting information unique to SPF-PFS program grantees that is otherwise not available to project officers or the PEP-C Cross site evaluation team. A literature review prepared by the evaluation team in November 2013 confirmed that the information being collected by the PEP-C SPF-PFS MRT cannot be obtained through other sources.

5. Involvement of Small Entities

Participation in this data collection will not impose a significant impact on small entities. SPF-PFS grantees will usually consist of State agencies, tribal organizations and other jurisdictions. Some subrecipients may be small entities; however, the SPF-PFS Performance Monitoring system is designed to include only the most pertinent information needed to be able to monitor the grantee’s progress and to carry out the evaluation effectively, and their impact will not be significant.

6. Consequences If Information Collected Less Frequently

The multiple data collection points for the SPF-PFS MRT are necessary to track and evaluate grantees’ and community subrecipients’ progress and change over time. In addition to performance monitoring purposes, the PEP-C team will use the data for the purposes of the SPF-PFS evaluation, and grantee and subrecipient communities will use these data to track their ongoing implementation. Less frequent reporting will affect SAMHSA’s and the grantees’ ability to do so effectively. For example, SAMHSA’s federal requirements require them to report on performance and GPRA measures once each year. Related measures include items community outcomes. New federal health disparities priorities require periodic reports of the activities used to address those priorities through SAMHSA programs such as PFS.

SAMHSA has made every effort to ensure that data are collected only when necessary and that extraneous collection will not be conducted. For example, PFS grantees report only outcomes required for GPRA measures on an annual basis. Other community outcomes needed for evaluation purposes can be reported less frequently. In addition, while SPF-PFS progress reports are required quarterly to ensure ongoing monitoring, items related to grantee funding sources are only required once each year. The system provides grantees the ability to report on grantee outcomes to supplement those available through other data sources or provide grantee outcomes data when not available from those other sources (for jurisdiction and tribal grantees). Substitute data source requests are only required when grantees cannot provide annual community outcomes data through pre-approved measures. Exhibit 4 provides information on data collection requirements and timing for the instruments of the PEP-C SPF-PFS MRT.

Exhibit 4. Data Collection Requirements and Timing for the PEP-C MRT Instruments

Instrument	Requirement	Timing
Contact Information	Yes; grantee information pre-filled	Generally once at beginning of grant then as grantee staff, subrecipients, and subrecipient staff are added or information needs to be updated
Quarterly Progress Reports (QPR)	Yes	Quarterly (4 times each year)
QPR Funding Data	Yes	Once each year
Outcome Data
Grantee Target Outcome Data	Yes	Once at the beginning of the grant (updated if needed)
PFS Selected Grantee-Level Outcome Data	No; suggested for jurisdiction or tribal grantees or for grantees who want to provide supplemental data	Once per year or less
Community-Level Outcome Data for Subrecipients	Yes	Reports on two* outcomes required annually; reports on additional outcomes required for at least two time points
Substitute Data Source Request	Only if grantee wants to use an annual required measure that is not pre-approved**	Once for each substitute data source measure

* One additional annual outcome needed for each additional grantee targeted substance beyond underage drinking and prescription drug misuse.

**See the PEP-C PFS Community Outcomes Manual, tables 3 and 4 for pre-approved outcomes (see https://pep-c.rti.org/HERO/KB/PEP-C-KB/Content/PFS/Community%20Outcomes/CO-Guidance-Manual/Community-Outcomes-Guidance-Manual.pdf)

7. Consistency With the Guidelines in 5 CFR 1320.5(d)(2)

This information collection fully complies with the guidelines in 5 CFR 1320.5(d)(2).

8. Consultation Outside the Agency

The notice required by 5 CFR 1320.8(d) was published in the Federal Register on September 30, 2015 (80 FR 58741). No comments were received.

These program monitoring tools were based on the original narrative tools completed by previous SPF-SIG grantees. In addition, the tools were reviewed by SAMHSA staff and contractors. These experts provided feedback on each of the data collection instruments and the instruments were revised based on their feedback. Revisions ranged from changes in the instructions to simplify them, to the addition of a module on health disparities in the SPF-PFS quarterly progress report. See Exhibit 11 for the list of individuals consulted throughout the development process of the instruments.

9. Payment to Respondents

No cash incentives or gifts will be given to respondents.

10. Assurance of Confidentiality

The PEP-C PFS MRT only requests personal data through the Contact Information instrument. That staff role, name, e-mail, and telephone number data collected through that instrument are collected to allow PEP-C to provide grantees and subrecipients for login information for the MRT system, and to facilitate contact with the grantee and subrecipient staff on their data entry, data cleaning needs, and technical assistance requests. This identifying information will be accessible only to select PEP-C evaluation staff and PFS Project Officers at SAMHSA. No other personal information will be collected from respondents as the focus of the data collection is on the programmatic characteristics of the SPF-PFS grantees and subrecipients.

No individual-level or personal data will be collected through the Quarterly Progress Report, Grantee Target Outcome Data, PFS Selected Grantee-Level Outcome Data, Community-Level Outcome Data for Subrecipients, or Substitute Data Source Request instruments. Grantee staff will provide information about their organizations and their PFS activities, rather than information about themselves personally. The instruments collect programmatic data at the grantee and community levels along with aggregated, nonidentifying individual-level data (e.g., community outcomes data). Sensitive respondent information, such as birthdates and Social Security Numbers, will not be collected.

The PEP-C systems development team and the SAMHSA CDP development team take responsibility for ensuring that the Web and data system is properly maintained and monitored. Server staff will follow standard procedures for applying security patches and conducting routine maintenance for system updates. Data will be stored on a password-protected server, and access to data in the system will be handled by a hierarchy of user roles, with each role conferring only the minimum access to system data needed to perform the necessary functions of the role.

While not collecting individual-level data, contractor staff are trained on the importance of privacy and in handling sensitive data.

11. Questions of a Sensitive Nature

There are no questions of a sensitive nature in this collection.

12. Estimates of Annualized Hour Burden

The number of data collection respondents will vary by year because of the varying lengths in grants, data collection time points, and each cohort’s grant end dates. As such, the burden and respondent cost will also vary by year. Exhibit 5 provides an overview of the estimated annual number of responses per grantee, per instrument, broken out by cohorts.

Exhibit 5. Annual Data Collection Responses by Cohort per Grantee for PEP-C SPF-PFS MRT Instruments

	FY2016	FY2017	FY2018	FY2019–Request OMB Extensiona	FY2020–Request OMB Extensiona
Contact Information
PFS 2013	.25	.25	.25	0	0
PFS 2014	.25	.25	.25	.25	0
PFS 2015	1	.25	.25	.25	.25
Quarterly Progress Report
PFS 2013	4	4	4	0	0
PFS 2014	4	4	4	4	0
PFS 2015	4	4	4	4	4
Grantee Target Outcome Data
PFS 2013	0	0	0	0	0
PFS 2014	0	0	0	0	0
PFS 2015	1	0	0	0	0
PFS Selected Grantee-Level Outcome Data
PFS 2013	.06	.06	.06	0	0
PFS 2014	.24	.24	.24	.24	0
PFS 2015	0	.16	.16	.16	.16
Community Level Outcome Data for Subrecipients
PFS 2013	1	1	1	0	0
PFS 2014	1	1	1	1	0
PFS 2015	0	1	1	1	1
Substitute Data Source Request
PFS 2013	.25	0	0	0	0
PFS 2014	.52	.24	0	0	0
PFS 2015	0	.5	.25	0	0

Note. OMB, Office of Management and Budget; PFS, Partnerships for Success; SPF, Strategic Prevention Framework.

Note. PFS II grantees not included on this table as most of these grantees will have completed their grants, and all are also included in the PFS 2015 cohort.

^a FY2019 and FY2020 do not fall within the OMB 3-year approval period; therefore, data collection for those years is not included in the burden estimate.

Contact Information Instrument

All 69 PFS 2013 (n=16), PFS 2014 (n=21), and PFS 2015 (n=32) grantees, and all future cohorts, are expected to provide the data collected through the Contact Information Instrument at the beginning of their grant and then update that information as needed in future years. The Contact Information Instrument is estimated to take 1 hour to complete per response; this includes time to look up and compile information (0.5 hours) and time to complete the Web-instrument (0.5 hours). After the first year updates to the Contact Information Instrument are estimated to take only .25 hours to complete per response. The estimated burden time is based on test instruments completed by evaluation staff members that have experience working with SPF-PFS grantees (see Section B.4 for more detail). There are no direct costs to respondents other than their time to complete the instrument. Exhibits 6–8 provide the details of the annual burden for each instrument for FY2016–FY2018, and Exhibit 9 presents estimates of the Contact Information Instrument annualized burden hours, 25, and the annualized respondent cost, $2,820 (total burden hours × the average hourly wage for State government managers, as reported in the 2014 Occupational Employment Statistics [OES] by the Bureau of Labor Statistics [BLS]; see http://www.bls.gov/oes/current/naics4_999200.htm#11-0000).

Quarterly Progress Report

All 69 PFS 2013 (n=16), PFS 2014 (n=21), and PFS 2015 (n=32) grantees, and all future cohorts, are expected to complete Quarterly Progress Reports four times each year, beginning with the first quarter in their first year of the grant. The Quarterly Progress Report is estimated to take 3 hours to complete per response; this includes time to look up and compile information (2 hours) and time to complete the Web-instrument (1 hour). The estimated burden time is based on test instruments completed by evaluation staff members that have experience working with SPF-PFS grantees (see Section B.4 for more detail). There are no direct costs to respondents other than their time to complete the instrument. Exhibits 6–8 provide the details of the annual burden for each instrument for FY2016–FY2018, and Exhibit 9 presents estimates of the Quarterly Progress Report annualized burden hours, 828, and the annualized respondent cost, $33,849 (total burden hours × the average hourly wage for State government managers, as reported in the 2014 Occupational Employment Statistics [OES] by the Bureau of Labor Statistics [BLS]; see http://www.bls.gov/oes/current/naics4_999200.htm#11-0000).

Grantee Target Outcome Data

All 69 PFS 2013 (n=16), PFS 2014 (n=21), and PFS 2015 (n=32) grantees, and all future cohorts, are expected to complete Grantee Target Outcome Data Instruments one time during the first year of their grant. PFS 2013 and PFS 2014 grantees already provided this information, so only PFS 2015 grantees need to complete this instrument in FY2016. The Grantee Target Outcome Data Instrument is estimated to take .5 hour to complete per response; this includes time to complete the Web-instrument (grantees should not have to track down information for this instrument). The estimated burden time is based on test instruments completed by evaluation staff members that have experience working with SPF-PFS grantees (see Section B.4 for more detail). There are no direct costs to respondents other than their time to complete the instrument. Exhibits 6–8 provide the details of the annual burden for each instrument for FY2016–FY2018, and Exhibit 9 presents estimates of the Grantee Target Outcome Data Instrument annualized burden hours, 11, and the annualized respondent cost, $450 (total burden hours × the average hourly wage for State government managers, as reported in the 2014 Occupational Employment Statistics [OES] by the Bureau of Labor Statistics [BLS]; see http://www.bls.gov/oes/current/naics4_999200.htm#11-0000).

PFS Selected Grantee-Level Outcome Data

The PFS Selected Grantee-Level Outcome Data instrument will not be required of any grantees. Generally jurisdiction and tribal grantees will be more likely to choose to complete this instrument, though state grantees will also be able to complete it. We expect that 1 PFS 2013 grantee, 5 PFS 2014 grantees, 5 PFS 2015 grantees, and a portion of the grantees in all future cohorts, will complete PFS Selected Grantee-Level Outcome Data instrument one time each year, beginning in the second year of their grant. Therefore PFS 2015 grantees will not need to complete this instrument in FY 2016. The PFS Selected Grantee-Level Outcome Data instrument is estimated to take 1 hour to complete per response; this includes time to look up and compile information (.5 hour) and time to complete the Web-instrument (.5 hour). The estimated burden time is based on test instruments completed by evaluation staff members that have experience working with SPF-PFS grantees (see Section B.4 for more detail). There are no direct costs to respondents other than their time to complete the instrument. Exhibits 6–8 provide the details of the annual burden for each instrument for FY2016–FY2018, and Exhibit 9 presents estimates of the PFS Selected Grantee-Level Outcome Data instrument annualized burden hours, 9, and the annualized respondent cost, $368 (total burden hours × the average hourly wage for State government managers, as reported in the 2014 Occupational Employment Statistics [OES] by the Bureau of Labor Statistics [BLS]; see http://www.bls.gov/oes/current/naics4_999200.htm#11-0000).

Community Level Outcome Data for Subrecipients

All 69 PFS 2013 (n=16), PFS 2014 (n=21), and PFS 2015 (n=32) grantees, and all future cohorts, are expected to complete Community Level Outcome Data for Subrecipients instrument one time each year, beginning in the second year of their grant. Therefore PFS 2015 grantees will not need to complete this instrument in FY 2016. The Community Level Outcome Data for Subrecipients instrument is estimated to take 3 hours to complete per response; this includes time to look up and compile information (2 hours) and time to complete the Web-instrument (1 hour). The estimated burden time is based on test instruments completed by staff members that have experience working with SPF-PFS grantees (see Section B.4 for more detail). There are no direct costs to respondents other than their time to complete the instrument. Exhibits 6–8 provide the details of the annual burden for each instrument for FY2016–FY2018, and Exhibit 9 presents estimates of the Community Level Outcome Data for Subrecipients instrument annualized burden hours, 175, and the annualized respondent cost, $7,154 (total burden hours × the average hourly wage for State government managers, as reported in the 2014 Occupational Employment Statistics [OES] by the Bureau of Labor Statistics [BLS]; see http://www.bls.gov/oes/current/naics4_999200.htm#11-0000).

Substitute Data Source Request

The Substitute Data Source Request instrument is only required of grantees if they want to use an annual required measure in their community outcome reporting that is not pre-approved. Usually grantees make these requests in the second or third years of their grants, with most requests occurring shortly before they need to report baseline community outcomes in their second grant year. In FY 2016 we expect that 4 PFS 2013 grantees, 11 PFS 2014 grantees, and 0 PFS 2015 grantees will complete the Substitute Data Source Request instrument. In FY 2017 we expect that 0 PFS 2013 grantees, 5 PFS 2014 grantees, and 16 PFS 2015 grantees will complete the Substitute Data Source Request instrument. In FY 2018 we expect that 0 PFS 2013 grantees, 0 PFS 2014 grantees, and 8 PFS 2015 grantees will complete the Substitute Data Source Request instrument. The Substitute Data Source Request instrument is estimated to take 1 hour to complete per response; this includes time to look up and compile information (.5 hour) and time to complete the Web-instrument (.5 hour). The estimated burden time is based on test instruments completed by staff members that have experience working with SPF-PFS grantees (see Section B.4 for more detail). There are no direct costs to respondents other than their time to complete the instrument. Exhibits 6–8 provide the details of the annual burden for each instrument for FY2016–FY2018, and Exhibit 9 presents estimates of the Substitute Data Source Request instrument annualized burden hours, 15, and the annualized respondent cost, $613 (total burden hours × the average hourly wage for State government managers, as reported in the 2014 Occupational Employment Statistics [OES] by the Bureau of Labor Statistics [BLS]; see http://www.bls.gov/oes/current/naics4_999200.htm#11-0000).

Exhibit 6. FY2016 Annual Burden

Instrument	Number of Respondents	Responses per Respondent	Total Number of Responses	Hours per Response	Total Burden Hours	Average Hourly Wage	Total Respondent Costa
Contact Information	69	1	69	1	69	$40.88	$2,820
Quarterly Progress Report	69	4	276	3	828	40.88	$33,849
Grantee Target Outcome Data b	32	1	32	.5	16	40.88	$654
PFS Selected Grantee-Level Outcome Datac	6	1	6	1	6	40.88	$245
Community Level Outcome Data for Subrecipientsc	37	1	37	3	111	40.88	$4,537
Substitute Data Source Requestc	15	1	15	1	15	40.88	$614
FY2016 Total	228		435		1,017		$42,719

^a Total respondent cost is calculated as total burden hours x average hourly wage.

^b PFS 2013 and PFS 2014 cohorts grantees provided this information prior to FY2015

^c PFS 2015 Grantees are not required to provide grantee or community outcomes data until FY2017

Exhibit 7. FY2017 Annual Burden

Instrument	Number of Respondents	Responses per Respondent	Total Number of Responses	Hours per Response	Total Burden Hours	Average Hourly Wage	Total Respondent Costa
Contact Information	69	1	69	1	69	$40.88	$2,820
Quarterly Progress Report	69	4	276	3	828	$40.88	$33,849
Grantee Target Outcome Data	--	--	--	--	--	--	--
Selected Grantee-Level Outcome Data	11	1	11	1	11	$40.88	$450
Community Level Outcome Data	69	1	69	3	207	$40.88	$8,462
Substitute Data Source Request	21	1	21	1	21	$40.88	$858
FY2017 TOTAL	239		446		1,136		$46,439

^a Total respondent cost is calculated as total burden hours x average hourly wage.

Exhibit 8. FY2018 Annual Burden

Instrument	Number of Respondents	Responses per Respondent	Total Number of Responses	Hours per Response	Total Burden Hours	Average Hourly Wage	Total Respondent Costa
Contact Information	69	1	69	1	69	$40.88	$2,820
Quarterly Progress Report	69	4	276	3	828	$40.88	$33,849
Grantee Target Outcome Data	--	--	--	--	--	--	--
Selected Grantee-Level Outcome Data	11	1	11	1	11	$40.88	$450
Community Level Outcome Data	69	1	69	3	207	$40.88	$8,462
Substitute Data Source Request	8	1	8	1	8	$40.88	$327
FY2018 TOTAL	226		433		1,123		$45,908

^a Total respondent cost is calculated as total burden hours x average hourly wage.

Exhibit 9. Annualized Data Collection Burden^a

Annualized Data Collection Burden

Instrument	Number of Respondents	Responses per Respondent	Total Number of Responses	Burden hours per Response	Total Burden Hours	Average Hourly Wage	Total Respondent Costa
Contact Information	69	1	69	1	69	40.88	$2,820
Quarterly Progress Report	69	4	276	3	828	40.88	$33.849
Grantee Target Outcome Data	11	1	11	1	11	40.88	$450
Selected Grantee-Level Outcome Data	9	1	9	1	9	40.88	$368
Community Level Outcome Data	58	1	58	3	175	40.88	$7,154
Substitute Data Source Request	15	1	15	1	15	40.88	$613
OVERALL TOTAL	69		438		1,107		$45,254

^a Annualized Data Collection Burden captures the average number of respondents and responses, burden hours, and respondent cost over the three years (FY2016 – 2018).

^b Respondent cost is calculated as total burden hours x average hourly wage.

13. Estimates of Annualized Cost Burden to Respondents

There are no respondent costs for capital or start-up or for operation or maintenance.

14. Estimates of Annualized Cost to the Government

The total estimated cost to the government for the data collection from FY 2016 through FY 2018 is $2,086,153. This includes approximately $1,602,547 for developing the instruments; programming and maintaining the online data collection system; providing data collection training to grantees and subrecipients; processing, cleaning, and housing data; and analyzing and reporting data. Approximately $55,602 per year represents SAMHSA costs to manage/administer the data collection and analysis for 25% each of two employees (GS-14-10, $111,203 annual salary). Approximately $105,600 per year represents SAMHSA costs to monitor and approve grantee reporting in these instruments (10% time of 10 Project Officers at $105,600 annual salary). The annualized cost is approximately $695,384.

15. Changes in Burden

This is a new collection of information.

16. Time Schedule, Publications, and Analysis Plan

Time Schedule

Exhibit 10 outlines the key time points for the PEP-C PFS MRT data collection.

Exhibit 10. Time Schedule for Data Collection

Activity	Time Schedule
Prepare for data collection, including programming Web system	November 2013–January 2016
Obtain OMB approval for data collection	January 2016
Collect data	January 2016–September 2018
Analyze data	April 2016–September 2018
Disseminate findings Interim reports, presentations, manuscripts, final report	April 2016–September 2018

Note. OMB, Office of Management and Budget; PFS, Partnerships for Success.

Publications

The PEP-C SPF-PFS evaluation will use the data collected through the PEP-C PFS MRT to help SAMHSA reach its diverse stakeholders through targeted products and innovative dissemination venues. The objective for all reports and dissemination products is to provide user-friendly documents and presentations that help SAMHSA successfully disseminate and explain the findings. The dissemination plan includes products in a variety of formats for a variety of target audiences. Audiences for these reports will include Congress, the ONDCP, SAMHSA Centers, the evaluation’s SAMHSA Contracting Officer’s Representatives (CORs), SPF-PFS grantees, and the broader substance abuse prevention field (e.g., academia, researchers, policy-makers, providers). PEP-C and SAMHSA recognize that different audiences are best reached by different types of report formats. For example, reports to Congress and ONDCP will require materials that are concise but offer policy-relevant recommendations. Reports created for SAMHSA Centers and the CORs will require more in-depth information, such as substantive background and discussion sections, to supplement the analytic approach. Reports created for SPF-PFS grantees will be concise handouts with helpful and easy-to-read graphics on performance data rather than lengthy text. The assortment of disseminations products developed using the PEP-C PFS MRT data will include short and long analytic reports, congressional briefings, annual evaluation reports, research and policy briefs, ad hoc analytic reports, journal articles, best practice summaries, and conference or other presentations.

Analysis

The PEP-C SPF-PFS evaluation uses a series of interdependent analysis frameworks that have been selected to maximize the coverage of key evaluation questions (EQs) posed for assessing the objectives of SPF-PFS in the prevention of onset and the reduction of the progression of UAD and PDM and their consequences. The evaluation will fully incorporate all data from the PEP-C PFS MRT as indicated in Exhibit 3. The analysis plan proposes a series of analyses that move from basic descriptive analyses of GPRA measures, grantee performance measures, and NOMs measures (e.g., means, frequencies, percentages) to the use of sophisticated qualitative analysis techniques and multiple analytic frameworks that reflect various complexities that are anticipated to arise with data collected by the PEP-C.

Matched Comparison Groups

The SPF-PFS evaluation will use a pre/post design with matched comparison groups where relevant and possible. The PEP-C team plans to obtain relevant baseline census, archival, and survey estimates to select comparison counties (or communities) for SPF-PFS subrecipients. For some grantees, much of the required estimates will be available through standard (public) reporting; for others, the PEP-C team will need to collaborate with grantee-level evaluators to obtain the estimates. In no cases will new data collection be required for the matching process and the follow-up outcomes data for the matched comparison groups will come from the same data sources as those used for the matching process.

Matched comparison communities will not be completing any of the instruments in the PEP-C PFS MRT.

Qualitative Analyses

Qualitative analyses of the PEP-C PFS MRT data focus primarily on open ended responses grantees provide to describe their SPF step accomplishments and barriers. PEP-C staff will upload the open-ended response data into a qualitative research software program, NVivo, for coding. Preparation for coding will include developing a dictionary or codebook in which codes will be carefully defined and logged so that coders are able to follow their meaning and know when to apply the codes to text within an interview. Codes will reflect prominent themes relevant to interpreting evaluation findings. To ensure reliability in the coding process, coders will then be assigned to work independently and concurrently on a subset of the open-ended response data. A kappa coefficient of .8 or higher will be maintained on all codes. Any discrepancies will be worked out between coders to ensure consistent application of codes. Upon completion of coding, the findings will be compiled on the basis of the prominence of codes (or themes) and organized around the major research questions and constructs. The findings that emerge will be used to examine grantee progress through the SPF steps along with relationships between that progress and outcomes.

Qualitative comparative analysis. The SPF-PFS evaluation plans to use qualitative comparative analysis (QCA). QCA is a case-oriented approach that examines relationships between conditions (similar to explanatory variables in regression models) and an outcome using set theory, a form of logic that deals with the nature and relations of sets. While few methodological approaches can accommodate the small number of grantees, QCA is a method designed for studies with small and intermediate numbers (i.e., 10 to 50 cases). QCA examines what conditions—alone or in combination with other conditions—are necessary or sufficient to produce an outcome; in contrast, regression analyses identify “what factor, holding all other factors constant at each factor’s average, will increase (or decrease) the likelihood of an outcome.” Because of the intermediate number of grantees (n=69 total PFS 2013, PFS 2014, and PFS 2015 grantees), QCA will allow us to explore EQs for the intermediate-number populations when probabilistic analysis may not be possible.

We plan to use data from the Quarterly Progress Reports to operationalize “conditions sets” (similar to independent variables in regression) and Community Outcome data to operationalize “outcomes sets” (similar to dependent variables). We will abstract the relevant values from appropriate data sources and create a Stata 13 data set. We will follow conventional QCA practices, which include identifying individual necessary and sufficient conditions, examining the combinations of conditions (i.e., sufficient causal pathways), and assessing QCA parameters of fit (i.e., consistency and coverage).

Quantitative Analyses

Several features of the evaluation design and EQs guided the selection of the analysis frameworks, including:

• Repeated outcomes;

• Data from subrecipient communities nested within grantees;

• Nonrandomized comparison communities within grantee States; and

• Nonrandom selection of intervention types that often occurr in combination

Each of these features led to the selection of the complex analysis frameworks the SPF-PFS evaluation has proposed to use or adapt. Below is an overview of the more advanced analytic frameworks that will be used in the SPF-PFS evaluation, which include:

• multilevel latent growth models (MLLGMs, with parallel and lagged processes)

• integrative data analysis/item response theory

• meta-regression

• propensity score weighting

Multilevel latent growth models. One of the primary analysis frameworks that will be used is the MLLGM. The basic linear MLLGM (Muthén, 1997) is constructed to account for variability in changes over time on outcomes, with sources of variability at the grantee and subrecipient levels. Where possible, a multiple baseline strategy will be employed whereby trends over time on outcomes at the grantee- and sub-recipient levels prior to PFS implementation will be compared to post-implementation trends (similar to an interrupted time series approach). In addition, predictors of post-implementation changes in outcomes over time, such as the type and dosage of interventions supported under SPF-PFS and variation in outcomes across SPF-PFS cohorts (i.e., PFS 2013, PFS 2014, and PFS 2015), will be the focus of these analyses. However, several limitations may arise in these analyses including (1) small sample sizes at the grantee level, (2) nonrandom assignment of PFS interventions, and (3) variation in how GPRA and NOMs may be reported within and across grantees. As a result, the SPF-PFS evaluation will incorporate alternative or complementary analysis frameworks, or both, in addition to MLLGM.

Integrative data analysis. To address concerns about the potential variability in measures across grantees, the SPF-PFS evaluation will employ integrative data analysis (Curran et al., 2008; Curran & Hussong, 2009) to harmonize different measures of UAD and PDM (as well as risk and consequences measures) across grantees and subrecipients. The harmonization process involves (1) creating a common measure for questions that are worded slightly differently from each other but are comparable and (2) using response scales (e.g., Likert-type scales, ordered categories) that can be condensed to their least common denominator (e.g., ever used/never used). For single-item constructs and measures, the harmonization process is the only step necessary. For constructs that reflect multiple-item scales, confirmatory factor analysis models will be employed to assess which items load on which factors and derive factor and scale scores via item response theory models, which weight each item according to how common (or rare) a response is and how correlated the item is with other items making up the factor. Note that this step may be more difficult at the grantee level, where sample sizes are small.

Meta-regression. A second strategy that can be employed if sample sizes are too small to estimate MLLGMs or too small to estimate scale scores under integrative data analysis is meta-regression (Hox, 2010). Meta-regression uses effect sizes for data instead of raw data (as is done with meta-analysis, where effect sizes are extracted from journal articles). Unlike MLLGMs, meta-regression does not require that the outcome measure be exactly the same across all analysis units; effect sizes for changes over time from disparate measures of the same construct within grantee (for grantee-level analyses) are sufficient for analysis. In addition to effect sizes, the standard errors for the effect sizes are used to calculate meta-regression weights in a manner similar to that of standard meta-analysis models. Key predictors can then be used to account for variability in effect sizes as in a standard meta-analysis.

Propensity scoring approaches. Propensity scoring is a statistical approach used to balance measured covariates that influence the probability of selection into two or more non-experimental groups and also influence treatment outcomes (Rosenbaum & Rubin, 1983; Shadish, Cook, & Campbell, 2002; West, Biesanz, & Pitts, 2000); more recent work has extended propensity scoring to continuous measures of treatment (Imai & van Dyk, 2004). The propensity score (when treatment assignment is categorical) is the predicted probability of assignment to a treatment condition given the key covariates of interest (estimated from a regression model—ordinary least squares for continuous treatment or logistic for categorical treatment), with the resulting probability used as either a sample stratifier or a weight in subsequent outcome analyses. After the propensity score weight is controlled for, covariate distributions should be equal across conditions, which will mimic random assignment to the conditions of interest in the particular EQ. These scores can then be used to weight outcome analyses (e.g., MLLGMs) to produce unbiased estimates of the treatment effect (Harder, Stuart, & Anthony, 2010; McCaffrey, Ridgeway, & Morral, 2004; Rosenbaum & Rubin, 1983; Shadish, 2010).

17. Display of Expiration Date

OMB approval expiration dates will be displayed.

18. Exceptions to Certification for Statement

There are no exceptions to the certification statement. The certifications are included in this submission.

REFERENCES

Calear, A. L., & Christensen, H. (2010). Systematic review of school-based prevention and early intervention programs for depression. Journal of Adolescence, 33(3), 429–438.

Centers for Disease Control and Prevention. (2012). CDC grand rounds: prescription drug overdoses - a U.S. epidemic. MMWR. Morbidity and Mortality Weekly Report, 61(1), 10–13.

Cheong, J., MacKinnon D. P., & Khoo S.-T. (2003). Investigation of mediational processes using parallel process latent growth curve modeling. Structural Equation Modeling, 10, 238–262.

Curran, P. J., & Hussong, A. M. (2009). Integrative data analysis: the simultaneous analysis of multiple data sets. Psychological Methods, 14(2), 81–100.

Curran, P. J., Hussong, A. M., Cai, L., Huang, W., Chassin, L., Sher, K. J., & Zucker, R. A. (2008). Pooling data from multiple longitudinal studies: The role of item response theory in integrative data analysis. Developmental Psychology, 44, 365–380.

Drummond, M. F., Sculpher, M. J., Torrance, G. W., O’Brien, B. J., & Stoddart, G. L. (2005). Methods for the economic evaluation of health care programmes (3rd ed.). Oxford, UK: Oxford University Press.

Feinberg, M. E., Greenberg, M. T., Osgood, D. W., Sartorius, J., & Bontempo, D. (2007). Effects of the Communities That Care model in Pennsylvania on youth risk and problem behaviors. Prevention Science, 8(4), 261–270.

Gold, M. R., Siegel, J. E., Russell, L. B., & Weinstein, M. C. (Eds.). (1996). Cost-effectiveness in health and medicine. New York, NY: Oxford University Press.

Harder, V. S., Stuart, E. A., & Anthony, J. (2010). Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research. Psychological Methods, 15, 234–249.

Hox, J. J. (2010). Multilevel analysis: Techniques and applications (2nd ed.). New York, NY: Routledge.

Imai, K., & van Dyk, D. A. (2004). Causal inference with general treatment regimes: Generalizing the propensity score. Journal of the American Statistical Association, 99, 854–866.

Jagers, R. J., Morgan-Lopez, A., Flay, B., & Aban Aya Investigators. (2009). Age and intervention effects on youth violent behavior. Journal of Primary Prevention, 30, 642–658.

Johnston, L. D., O’Malley, P. M., Bachman, J. G., & Schulenberg, J. E. (2013). Monitoring the Future national results on drug use: 2012 overview, key findings on adolescent drug use. Ann Arbor: Institute for Social Research, The University of Michigan.

Lemstra, M., Bennett, N., Nannapaneni, U., Neudorf, C., Warren, L., Kershaw, T., et al. (2010). A systematic review of school-based marijuana and alcohol prevention programs targeting adolescents aged 10–15. Addiction Research and Theory, 18, 84–96.

McCaffrey, D. F., Ridgeway, G. and Morral, A. R. (2004). Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychological Methods, 9, 403–425.

Miller, T. R., Levy, D. T., Spicer, R. S., & Taylor, D. M. (2006). Societal costs of underage drinking. Journal of Studies on Alcohol, 67(4), 519–528.

Muthén, B. (1997). Latent variable modeling with longitudinal and multilevel data. In A. Raftery (Ed.), Sociological Methodology (pp. 453–480). Boston, MA: Blackwell Publishers.

National Institutes of Health. (2011, October). Prescription drugs: Abuse and addiction (NIH Publication No. 11-4881). Retrieved from http://www.drugabuse.gov/sites/default/files/rrprescription.pdf

Office of Juvenile Justice and Delinquency Prevention. (2012, September). Effects and consequences of underage drinking (NCJ 237145). Juvenile Justice Bulletin. Washington, DC: Author. Retrieved from http://www.ojjdp.gov/pubs/237145.pdf

Office of National Drug Control Policy. (2011). Epidemic: responding to America’s prescription drug abuse crisis. Retrieved from: http://www.whitehouse.gov/sites/default/files/ondcp/issues-content/prescription-drugs/rx_abuse_plan_0.pdf

Office of the Surgeon General. (2007). The Surgeon General’s call to action to prevent and reduce underage drinking. Rockville, MD: Author. Retrieved from: http://www.surgeongeneral.gov/library/calls/underagedrinking/calltoaction.pdf

Orwin, R. G., Stein-Seroussi, A., Edwards, J. M., Landy, A. L., & Flewelling, R. L. (2014). Effects of the Strategic Prevention Framework State Incentives Grant (SPF SIG) on state prevention infrastructure in 26 states. Journal of Primary Prevention, 35, 163-180.

Piper, D., Stein-Seroussi, A., Flewelling, R., Orwin, R. G., & Buchanan, R. (2012). Assessing state substance abuse prevention infrastructure through the lens of CSAP’s Strategic Prevention Framework. Evaluation and Program Planning, 35(1), 66–77.

Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.

Sacks, J. J., Roeber, J., Bouchery, E. E., Gonzales, K., Chaloupka, F. J., & Brewer, R. D. (2013). State costs of excessive alcohol consumption, 2006. American Journal of Preventive Medicine, 45(4), 474–485.

Shadish, W., Cook, T., & Campbell, D. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin.

Shadish, W. R. (2010). Campbell and Rubin: A primer and comparison of their approaches to causal inference in field settings. Psychological Methods, 15, 3–17.

Spoth, R., Greenberg, M., Bierman, K., & Redmond, C. (2004). PROSPER community-university partnership model for public education systems: capacity-building for evidence-based, competence-building prevention. Prevention Science, 5(1), 31–39.

Spoth, R., Greenberg, M., & Turrisi, R. (2008). Preventive interventions addressing underage drinking: State of the evidence and steps toward public health impact. Pediatrics, 121(Suppl 4), S311–336.

Spoth, R., Greenberg, M., & Turrisi, R. (2009). Overview of preventive interventions addressing underage drinking: State of the evidence and steps toward public health impact. Alcohol Research & Health, 32(1), 53–66.

Spoth, R., Redmond, C., Clair, S., Shin, C., Greenberg, M., & Feinberg, M. (2011). Preventing substance misuse through community-university partnerships: randomized controlled trial outcomes 4(1/2) years past baseline. American Journal of Preventive Medicine, 40(4), 440–447.

Substance Abuse and Mental Health Services Administration. (2012a). Results from the 2011 National Survey on Drug Use and Health: Summary of national findings (NSDUH Series H-44, HHS Publication No. (SMA) 12-4713). Rockville, MD: Author. Retrieved from http://www.samhsa.gov/data/NSDUH/2k11Results/NSDUHresults2011.pdf

Substance Abuse and Mental Health Services Administration. (2012b). National cross-site evaluation of the Strategic Prevention Framework State Incentive Grants for Cohort III, IV, and V: Final evaluation report.

Substance Abuse and Mental Health Services Administration. (2012c). 2011 National Survey on Drug Use and Health: Sample design report. Retrieved from http://www.samhsa.gov/data/2k12/NSDUH2011MRB/NSDUHmrbSampleDesign2011.pdf.

Substance Abuse and Mental Health Services Administration. (2013a). Drug Abuse Warning Network, 2011: National estimates of drug-related emergency department visits. (DAWN Series D-39, HHS Publication No. (SMA) 13-4760). Rockville, MD: Author. Retrieved from http://www.samhsa.gov/data/2k13/DAWN2k11ED/DAWN2k11ED.htm#5

Substance Abuse and Mental Health Services Administration. (2013b). Results from the 2012 National Survey on Drug Use and Health: Summary of national findings (NSDUH Series H-46, HHS Publication No. (SMA) 13-4795). Rockville, MD: Author. Retrieved from http://www.samhsa.gov/data/NSDUH/2012SummNatFindDetTables/NationalFindings/NSDUHresults2012.pdf

Ttofi, M. M., & Farrington, D. P. (2011). Effectiveness of school-based programs to reduce bullying: A systematic and meta-analytic review. Journal of Experimental Criminology, 7, 27–56.

United States Department of Health and Human Services. (2011). National prevention strategy: America’s plan for better health and wellness. Washington, DC: National Prevention Council. Retrieved from: http://www.surgeongeneral.gov/initiatives/prevention/strategy/report.pdf

Wandersman, A., Duffy, J., Flaspohler, P., Noonan, R., Lubell, K., Stillman, L., et al. (2008). Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation. American Journal of Community Psychology, 41(3-4), 171–181.

West, S. G., Biesanz, J. C., & Pitts, S. C. (2000). Causal inference and generalization in field settings: Experimental and quasi-experimental designs. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (pp. 40–84). New York, NY: Cambridge University Press.

13