NOMs_OMB_SUPPORTING_STATEMENT_revised_090407MYnotrack416

SUPPORTING STATEMENT

National Outcome Measures for Substance Abuse Prevention (NOMs)

1. Justification

1. Circumstances of Information Collection

The Substance Abuse and Mental Health Services Administration's (SAMHSA) Center for Substance Abuse Prevention (CSAP) is requesting an revision of Office of Management and Budget (OMB) approval for CSAP's data collection set of National Outcome Measures (NOMs) identified for the field of prevention. The current approval, under OMB No. 0930-0230, is expiring on December 31, 2008. All new grantees initially funded at the end of FY08 and beyond (subject to OMB approval) will be required to use these measures as appropriate at the State, substate, program and participant levels. CSAP is requesting approval to continue collecting data using measures in the following domains: Abstinence from Alcohol and Other Drugs, Employment/Education, Crime and Criminal Justice, Access/Service Capacity, Retention, Social Support/Social Connectedness, Cost-Effectiveness, and Use of Evidence-Based Practices. These NOMs relate to youth ages 12 to 17 and to adults ages 18 and older.Depending on the measure and the targeted range of the grantee effort, data collection may only be required at the program, or sometimes at the community levels. These data already exist at the State levels and often at the substate level.

The authorization is Section 501 (d)(4) of the Public Health Service Act. The NOMS initiative emerged as a result of multiple, converging Federal events including the advent of GPRA, PART, the Substance Abuse and Mental Health Services Administration’s (SAMHSA) agency-wide cross-cutting GPRA tools in 1999 (OMB No. 0930-0208), the Office of National Drug Control Policy’s (ONDCP) Performance Measures of Effectiveness (PMEs), Healthy People 2010, legislation toward state performance partnership grants, and the general emphasis for accountability at all levels. As a consequence, the NOMs includes GPRA, and HP2010 measures and further supports these efforts by promoting common measures in the prevention field.

CSAP believes that NOMs measures are necessary to assess the performance of its prevention programs. Based on its long history of working with States, communities, and prevention providers, the Data Analysis Coordination and Consolidation Center (DACCC) and outside expert panels believe consistent prevention measures allow for valid comparison evaluations. CSAP is requesting to modify the wording of 12 previously approved questions in order to make them comparable to individual NOMs items. For example, NSDUH items on 30-day use ask respondents to report the number of days on which they used specific substances. Three currently approved NOMs 30-day use questions ask respondents for the number of occasions on which they used substances. CSAP would like to change the wording of these questions and their corresponding response options to conform to NSDUH wording. Second, response options for NSDUH questions typically include a Don't Know response option. CSAP is requesting modification of nine currently approved NOMs questions to include this response option.

CSAP believes that these measures are necessary to include as NOMs based on its long history working with states, communities and prevention providers, and on input from its Data Coordinating Center and outside expert panels who made recommendations based on a review of existing measures using standard criteria. Additionally, CSAP believes that these measures can be collected at the national, state, substate and program level, providing the consistency of measurement towards which Federal accountability efforts strive. The measures will be used as follows:

National/State: CSAP pre-populates State level NOMs measures for all but three domains using data from the National Survey on Drug Use and Health, the National Center for Education Statistics, National Highway Traffic Safety Administration Fatality Analysis Reporting System, and the FBI’s Uniform Crime Reporting Program. States supply the data on the number of persons served, cost efficiency, and evidence based practices from their own administrative databases. Outcome trend data for all of the State level NOMs are necessary to identify need and monitor global effectiveness at the population level in order to inform federal resource allocation decisions

Discretionary Grants: SAMHSA's CSAP has identified specific outcome measures that are required of non-State discretionary grant recipients. These NOMs represent the domains noted above and relate to youth ages 12 to 17 and to adults ages 18 and older. Grantees providing services are required to administer surveys to all participants at program entry (baseline), program exit, and three to six months following program exit.

The mission of SAMHSA’s Center for Substance Abuse Prevention (CSAP) is to decrease substance use and abuse and related problems among the American public. SAMHSA/CSAP accomplishes this through field-testing scientifically defensible programs; disseminating comprehensive, culturally appropriate prevention strategies, policies, and systems; and building capacity in states and community-based providers. Data are collected from SAMHSA/CSAP grants and contracts where participant outcomes are assessed. The analysis of these data helps determine whether progress is being made in achieving SAMHSA/CSAP’s mission.

The primary purpose of this proposed data activity is to promote the use among SAMHSA/CSAP grantees and contractors of common National Outcome Measures recommended by SAMHSA/CSAP with significant input from panels of experts and state representatives. The activity will be reorganized to reflect and support SAMHSA’s National Outcome Domains and therefore, ‘‘the National Outcome Measures for Substance Abuse Prevention’’ (NOMs) is the new title proposed for this activity. The use of consistent measurement for specified outcomes across SAMHSA/CSAP-funded projects will improve SAMHSA/CSAP’s ability to respond to the Government Performance and Results Act (GPRA), the Office of Management and Budget Program Assessment Rating Tool (PART) evaluation, and address goals and objectives outlined in the Office of National Drug Control Policy’s Performance Measures of Effectiveness.

2. Purpose and Use of Information

The primary use for these data will be to continue to strengthen accountability by standardizing and improving the quality of the data collected. SAMHSA/CSAP would like to further improve its ability to address questions from Congress and the public regarding what types of prevention programs work and for whom. Through the use of common, psychometrically sound measures across its many grants, SAMHSA/CSAP is able to compare results and, conduct analyses across data sets. The NOMs reduces the number of measures to those most critical to prevention and provides the best items for capturing those constructs as defined by experts in the field. With approval of this extension, SAMHSA/CSAP will continue to look across all of its programs to determine which ones are having the greatest impact on prevention outcomes. This type of cross-program information is vital to understanding the types of interventions that work, for which populations, and under what circumstances. Such information informs SAMHSA/CSAP’s future budget allocation and program development decisions, and allows SAMHSA/CSAP to address various requests and queries.

In addition to aiding SAMHSA/CSAP in responding to requests and queries, the NOMs include measures consistent with or comparable to four of the five SAMHSA/CSAP’s GPRA measures: 30 day use, age of first use, disapproval, and perceived risk. Direct service grantees typically collect data from program participants at three time periods: baseline, exit and 6 month follow-up. Data from indirect service and infrastructure building grantees are typically provided annually. Data are both longitudinal and cross-sectional. The SAMHSA/CSAP Data Analysis Coordination and Consolidation Center (DACCC) will be responsible for data analysis across grantee sites, while individual grantees will be responsible for their own analyses.

SAMHSA/CSAP will use these data to fulfill PART evaluations, respond to GPRA requirements, and to further establish an ongoing monitoring system that ensures the most effective and cost-effective programming available to meet the goals of SAMHSA/CSAP and HHS.

CSAP is proposing to eliminate 22 of the 49 measures that received OMB clearance in 2005, to reduce reporting burden for grantees. CSAP also requests permission to make minor changes to the question wording and response categories for some of the remaining measures (see Change Table and NOMs Definitions in AB). Since the National Survey of Drug Use and Health (NSDUH) provides an economical extant source of data for NOMs measures at the State level, it is important that the NOMs conform to NSDUH question wording.

3. Use of Information Technology

It is anticipated that technical infrastructure and data management skills will vary across grantee sites. The burden is greatly reduced by the fact that the data collection process can be conducted by submitting electronic files. In many cases, programs can report all data online. To maximize data accuracy and reliability, online data entry tools are being designed for all programs.

Currently, the percent of grantees submitting data electronically varies by SAMHSA/CSAP program. In some programs, 100 percent of the grantees submit data electronically. In other programs this percentage is much lower. SAMHSA/CSAP has developed a web-based data entry tool to assist grantees in submitting their data electronically. This data entry tool, available through CSAP’s Services and Accountability Monitoring System (CSAMs) and along with the electronic submissions will reduce the burden on those grantees that do not have the capacity to submit large batch files. These tools will be made available to grantees through CSAP’s Services Accountability and Monitoring System (CSAMS) Web portal. The tools are designed to reflect the structure of the questionnaires, and to allow the entry of data from completed questionnaires directly into the system through the use of radio buttons corresponding to response options. The system automatically quantifies the selected response options and stores the numeric codes in a SQL server for subsequent extraction, cleaning, and analysis.

CSAMS is maintained by CSAP’s Data Information Technology Infrastructure Center (DITIC). The data entered online by grantees are periodically extracted by DITIC and transmitted in encrypted form to CSAP’s Data Analysis Coordination and Consolidation Center (DACCC) for cleaning, record linkage, and analysis. Grantees have two options for accessing the data they enter online. In the first option, grantees can download, in spreadsheet form, the raw data they have entered online as soon as it is submitted. Grantees can also access their data from the cleaned analysis files prepared by DACCC which are posted on CSAMS under password protection.

Grantees that prefer to create their own data files have the option of uploading complete data files to CSAMS. A grantee choosing this data submission option is required to use a standard codebook while preparing the data, thus ensuring that uploaded data files have the same numeric coding and variable naming conventions as the data entered using the online tools.

Grantees’ NOMs data will be submitted to CSAP’s Data Information Technology Infrastructure Center (DITIC) in May and November (or annually as appropriate), which in turn will submit data to the DACCC. The DACCC will use these data for secondary analysis to aid SAMHSA/CSAP in responding to GPRA, ONDCP, and other Federal reporting requirements, and to inform SAMHSA/CSAP policy and program planning.

4. Efforts to Identify Duplication

Currently the NOMs measures help fulfill the GPRA requirements by incorporating the necessary questions. At their initial meeting with the GPO, the new grantees, state coordinators and the GPO will review the NOMs forms to ensure a clear understanding of the measures that are required. It is up to the state and individual grantee whether other data are collected in their respective jurisdictions. Because SAMHSA is promoting the use of NOMs across all programs, this effort will result in less overlap and duplication, and substantially reduce the burden on grantees that result from data demands associated with individual programs.

5. Involvement of Small Entities

Some program grantees that will be collecting these data are considered small entities: state level grantees, such as those participating in the SPF-SIG, are not. The anticipated additional burden on small entities due to data collection is expected to be minimal since grants include funding for evaluation activities.

6. Consequences If Information Collected Less Frequently

Direct service grantees usually collect the required data (e.g. previously approved GPRA measures) from participants before services are initiated, at the end of services, and 6 months after services end. Discharge and follow-up data collections are necessary to generate outcome data. Infrastructure and indirect service grantees submit annually. Grantees submit NOMs data to coincide with GPRA and PART data submissions. SAMHSA/CSAP will require all grantees and contractors to submit data to the CSAP DITIC for cross-program analysis twice per year (May and November). These submission dates are necessary for SAMHSA/CSAP to provide the most current performance reporting for requirements such as GPRA and PART. If these data are submitted to SAMHSA/CSAP less frequently, SAMHSA/CSAP will not be able to meet such reporting requirements.

7. Consistency with the Guidelines in 5 CFR 1320.5(d) (2)

The collection of the NOMs data fully complies with the guidelines in 5 CFR 1320.5(d) (2).

8. Consultation Outside the Agency

The notice required by 5 CFR 1320.8(d) was published in the Federal Register on Monday, September 22, 2008 (Vol. 73, No. 184, pages 54601-54603). No comments were received.

In developing the original set of prevention NOMS, SAMHSA/CSAP sought consultation outside of the agency from experts in the field (including researchers, evaluators and state representatives). The SAMHSA/CSAP Data Coordination and Consolidation Center (DCC) also examined a variety of measures to identify the most relevant, available and statistically sound measures for each of the constructs. This was done in a manner similar to that done for the OMB approved Core Measures. The criteria used by the DCC included selecting NOMs measures that met thirteen criteria, including whether or not the instrument is in the public domain, the ease of scoring, and evidence of cultural appropriateness. The DCC and outside expert participants who participated in these processes may be found in Attachment B.

9. Payment to Respondents

No cash payment will be made to individual program participants from whom data will be collected. Although not a project requirement, some grantee organizations provide in-kind incentives to respondents (such as gift certificates from local vendors), for completing the evaluation study. The decision to provide incentives is left to the discretion of local sites.

10. Assurance of Confidentiality

As part of its grant application process, SAMHSA/CSAP requires that grantees describe the procedures they will use to ensure the privacy and protection of participant data. These include by whom and how data will be collected, how data collection instruments will be administered, where data will be stored, who will/will not have access to information, and how the identity of participants will be safeguarded. Data files provided by the grantees to the DITIC do not contain client identifiers. The DACCC reviews these data files to ensure identifiers are removed before creating analysis files.

11. Questions of a Sensitive Nature

It is necessary for grantees to ask participants questions of a sensitive nature, such as the use of alcohol or other drugs, in order to understand the needs of participants and to measure the impact of services. Grantees are required to have adequate consent procedures in place, and these procedures include obtaining and documenting active parental/guardian consent when necessary. SAMHSA review committees will not approve nor will SAMHSA fund a site without adequate provisions for meeting Federal policies regarding consent.

12. Estimates of Annualized Hour Burden

This request for OMB approval to collect a set of national outcome measures for prevention is an umbrella request; that is, this request covers all SAMHSA/CSAP grantees. The estimated annual hour burden is provided in the table below. This estimate is based on SAMHSA/CSAP’s projected new grantees for FY2009 through FY20011 that target individuals. Estimated hours are broken down by programs within divisions. (Note: SAMHSA/CSAP does not know the exact number of grantees beyond FY09). These estimates represent data collection primarily at the program level, although this may also be applicable for some community level measures. Indirect, larger programs such as the SPF SIG will be able to access data from publicly accessible datasets such as state and substate estimates from the UCR, FARS, DoEd, and NSDUH. Thus the data collection burden on them has been greatly reduced.

ESTIMATES OF ANNUALIZED COST BURDEN

SAMHSA/CSAP program	Number of grantees	Number of respondents	Responses per respondent	Hours/ response	Total hours	Hourly Wage	Total Hour Cost
FY 09
Science/Services:
Fetal Alcohol	23	4,800	3	0.75	10,800	$8.00	$86,400
Workplace	13	6,000	3	0.75	13,500	$8.00	$108,000
Capacity:
HIV/Targeted Capacity	135	35,300	3	0.75	79,425	$8.00	$635,400
SPF SIG	42		0
SPF SIG/Community Level *		480	1	0.75	360	$8.00	$2,880
SPF SIG/Program Level *		12,000	3	0.25	9,000	$8.00	$72,000
Methamphetamine	12	3,000	3	0.75	6,750	$8.00	$54,000
FY 10
Science/Services:
Fetal Alcohol	23	4,800	3	0.75	10,800	$8.00	$86,400
Workplace	13	6,000	3	0.75	13,500	$8.00	$108,000
Capacity:
HIV/Targeted Capacity	135	35,300	3	0.75	79,425	$8.00	$635,400
SPF SIG	42		0
SPF SIG/Community Level *		480	1	0.75	360	$8.00	$2,880
SPF SIG/Program Level *		12,000	3	0.25	9,000	$8.00	$72,000
Methamphetamine	12	3,000	3	0.75	6,750	$8.00	$54,000
FY 11
Science/Services:
Fetal Alcohol	23	4,800	3	0.75	10,800	$8.00	$86,400
Workplace	13	6,000	3	0.75	13,500	$8.00	$108,000
Capacity:
HIV/Targeted Capacity	135	35,300	3	0.75	79,425	$8.00	$635,400
SPF SIG	42		0
SPF SIG/Community Level *		480	1	0.75	360	$8.00	$2,880
SPF SIG/Program Level *		1,200	3	0.25	900	$8.00	$7,200
Methamphetamine	12	3,000	3	0.75	6,750	$8.00	$54,000
Annual Average		10,196		0.67	19,523	$8.00	$156,180

* The Strategic Prevention Framework State Incentive Grant (SPF SIG) has a three level evaluation: The Grantee, Community and Program Level. The Grantee level data will be pre-populated by SAMHSA. The use of the Community Level instrument is optional as they relate to targeted interventions implemented during the reporting period. At the program level, items will be selected in line with direct services implemented.

It is estimated that the average annual number of respondents over the three years will be 10,196. The average annual hour burden will be 19,523 hours. There will be no direct cost to youth or adults for participating in the study. The value of youth and adult time was assumed given the prevailing minimum wage rate in California, the State chosen since it is often the leading indicator for setting precedents later adopted by other States.

13. Estimates of Annualized Cost Burden to Respondents

Because data collection and reporting are already requirements for our grantees, there are no capital, start up, operational or maintenance service costs above what are current, usual practices in place. The CSAMS on-line data collection tool will be available at no cost to grantees.

14. Estimates of Annualized Cost to the Government

The estimate annualized cost to the government is 184,320 dollars. Use of the NOMs is part of the ongoing data collection and reporting activities SAMHSA/CSAP has planned and the Center has allocated sufficient resources for the efficient and effective management and use of the information to be collected. It is anticipated that the Government Project Officers who oversee the projects will expend time in assisting the DITIC and DACCC and grantees in appropriately responding to the measures. The GPO overseeing the DACCC will expend a portion of time overseeing the analysis of the NOMs data, as well as updating the measures as they are developed. It is anticipated that cross-program analysis will be conducted by the DACCC. Data analysis activities include processing the data received from the various programs, as well as conducting statistical analysis. These costs are broken out in the table below. Annual hours are based on a 40-hour work week for 48 weeks per year.

Estimated Annualized Cost to the Government

Position	Percent FTE	Annual Hours	Rate	Total Annual Cost
DACCC GPO	10%	192	$40	$7,680
DITIC GPO	5%	96	$40	$3,840
SPF SIG GPO	2.5%	48	$40	$1,920
HIV GPO	2.5%	48	$40	$1,920
Club/Meth GPO	2.5%	48	$40	$1,920
Workplace GPO	2.5%	48	$40	$1,920
FASD GPO	2.5%	48	$40	$1,920
DACCC Senior Analyst*	25%	480	$80	$38,400
DACCC Data Analyst*	50%	960	$65	$62,400
DACCC Data Manager*	50%	960	$65	$62,400
	Totals	2,928		$184,320

* Rate for DACCC staff includes fringe and overhead.

15. Changes in Burden

Currently there are 419 burden hours in the OMB inventory and CSAP is requesting 19,523 hours. The increase of 19,104 hours is due to changes in burden result from an increased number of grant programs and grantees using the NOMS measures. The following projects were not funded at the time of the original submission and have added a substantial number of grantees that are required to report NOMs: FASD, HIV Cohort 6 & 7, and SPF SIG Cohort 3 & 4.

16. Time Schedule, Publication and Analysis Plans

Publication and Analysis Plans. SAMHSA/CSAP will utilize the NOMs data on an ongoing basis to respond to GPRA and other Federal reporting requirements, as well as to address questions from Congress and the public regarding what types of prevention programs work and for which participants. Further, these data will be used to provide SAMHSA with data to document the overall agency performance requirements. The DACCC will conduct cross-program analyses and analyses of individual programs to provide information that will assist CSAP in planning and monitoring SAMHSA/CSAP’s success in meeting its goals. The table below provides examples of the various analyses and reports for which the NOMs data will be used, and when these activities take place.

Time Frame for Analytic Activities

Activity	Date
GPRA report to Congress	Yearly, in February
PART results for OMB	Varied and September yearly
Conducting analyses to support CSAP budget proposals to Congress	Spring of each year
Responding to ad hoc requests for analysis from CSAP staff, other Federal agencies, and the public regarding prevention effectiveness	Ongoing

The analyses of NOMs data will be descriptive, and, where suitable, inferential. Basic statistics will be calculated to derive frequency distributions, means, and other measures of central tendency. National outcome data will also be linked with participant demographic data to allow comparisons of program effectiveness between gender and other subpopulation groups.

17. Display of Expiration Date

The expiration date will be displayed.

18. Exceptions to Certification Statement

This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.

2. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

As part of its grant application, each grantee has an evaluation proposal, which specifies, among other things, the sampling methods to be used. Each evaluation proposal includes information on the response universe, sampling method, and the final sample sizes. These proposals are reviewed by a peer review group which assesses the adequacy and appropriateness of the study design and methods. Only those applicants having technically sound proposals are funded. In addition, SAMHSA/CSAP provides technical assistance as necessary to ensure that grantees carry out their evaluation as planned. SAMHSA/CSAP’s response universe for the NOMs renewal includes all grantees (with relevant participants and objectives) initially funded at the end of FY08 and beyond that provide direct services to participants.

2. Information Collection Procedures

Each grantee also has its own plan for data collection, processing, data cleaning, control, and retention. Each plan describes how uniform data collection will be ensured for the intervention and comparison groups, the time frame for conducting the assessments over the course of the project, and how participant protection will be assured. As mentioned above, these plans undergo peer review to ensure the adequacy and appropriateness of the study design and methods. The precise manner in which data will be collected and used depends on the specific grant program.

The SPF SIG program implements a five step process including needs assessment, building capacity, strategic planning, implementing evidence based programs practices and strategies, and evaluation. Subrecipient communities are given the option of using the Community NOMs Instruments to field sample surveys at regular intervals to track changes across time in the NOMs at the community level. Starting with Cohort 3 grantees, subrecipients implementing direct service programs will collect NOMs data at the participant level by administering the NOMs instruments at program entry, exit, and 6 months following exit.

The Fetal Alcohol Spectrum Disorder (FASD) Program integrates prevention and intervention approaches into existing service delivery systems as well as identifying and disseminating state of the art information for preventing FASD and increasing functioning and quality of life for those impacted by it.

Youth in the Workplace (YIW) identifies, adapts and implements effective prevention programs in the workplace for young adults. The collaborative has examined workplace response to substance abuse among youthful employees and studied emerging issues in order to develop policies related to the future needs of workplaces

The Minority Aids Initiative (MAI) develops local capacity to provide substance abuse prevention services for individuals living with and affected by HIV/AIDS. These funds assist with outreach and training, addressing the special needs of racial and ethnic minorities and studying the costs associated with delivering integrated care.

The Methamphetamine Program supports expansion of methamphetamine prevention interventions and/or infrastructure development. This program addresses the growing problem of methamphetamine abuse and addiction by assisting localities to expand prevention interventions that are effective and evidence-based and/or to increase capacity through infrastructure development. The goal is to intervene effectively to prevent, reduce or delay the use and/or spread of methamphetamine abuse.

These discretionary programs that include direct services to are surveyed at program entry, exit, and 6 months following exit.

Most measures are administered by pencil and paper. Other measures (e.g. number of alcohol related crashes) are derived from existing data sets (in this example, FARS). Grantees will submit data to the DITIC electronically, through CSAMS. CSAP has designed survey forms for the NOMS participant level measures. These are included in Attachment C.

Web-Based Data Entry Upload System: The DITIC has created CSAP’s Services Accountability & Monitoring System (CSAMS), an online data entry system that provides prevention information, data collection tools, documents, data entry functions, and access to reporting statistics and tracking. All of the questionnaires can be found in the “Tools” section of this Website. Common questionnaires are available in both Microsoft Word and PDF format for individual grant sites to download and make copies for administration to participants. Site evaluators or data collectors are expected to enter client or participant responses to questionnaires through the CSAMS Website. Sites will also be able to upload response databases through CSAMS that use the appropriate variable/value numbering (Questionnaire codebooks are also available on the “Tools” section of the CSAMS Website). SAMHSA/CSAP’s DACCC will be responsible for conducting logic checks on the data, and communicate with the grantees to clarify questions about the data.

3. Methods to Maximize Response Rates

As part of the terms and conditions of the grant award, sites are required to use the NOMs data system, available through the CSAMs website.

Issues related to response rates, as well as other data collection issues, are discussed at grantee meetings in order for GPOs to identify problems and provide technical assistance. In addition, GPOs monitor data collection efforts and provide technical assistance to individual grantees as necessary. Because collection of the NOMs is a stipulation of the grants, it is anticipated that all grantees will comply (as appropriate). The participants at each site to whom these measures will be administered are all voluntary respondents, and therefore grantees cannot guarantee full cooperation on the part of participants. Historically, however, participant response rates across grantee sites have averaged 80 percent.

4. Tests of Procedures

All measures on the NOMs are either part of the National Household Survey or are existing measures from existing databases. As a result, all measures are well tested and have been proven to be useful measures and no further pre-testing is needed.

5. Statistical Consultants

The measures submitted here for OMB approval renewal are a result of lengthy consultation and discussion among SAMHSA personnel, state representatives, the DCC staff and a panel of outside experts. Furthermore, these measures were presented and discussed with members of the grantee community, and at various meetings and conferences. The final selection of these measures was made by SAMHSA and CSAP senior officials. See Attachment B for staff and consultants involved.

Attachment Description

A NOMS Change Table and Definitions

B NOMS Review: Outside Experts

C NOMS Survey Forms (Adult Community, Youth Community, Adult Program, Youth Program)