Transformation Accountability (TRAC) Reporting System
Supporting Statement
The Substance Abuse and Mental Health Services Administration’s (SAMHSA), Center for Mental Health Services (CMHS) is requesting approval from the Office of Management and Budget (OMB) for a revision to continue the collection of performance data with current and future SAMHSA/CMHS grantees using the TRAC reporting system instruments (OMB No. 0930-0285) that expires on March 31, 2019.
SAMHSA/CMHS is requesting the revision of one of the current client services data collection instruments, as indicated below:
The CMHS NOMs Adult Client-level Measures for Discretionary Programs Providing Direct Services (Attachment 1). Revision: Adding 12 new recovery measures.
The CMHS NOMs Child Client-level Measures for Discretionary Programs Providing Direct Services (Child/Caregiver Version). No change.
The Infrastructure Development, Prevention, and Mental Health Promotion Performance Indicators. No change.
This information collection will allow SAMHSA to continue to meet the Government Performance and Results Act (GPRA) of 1993 reporting requirements that quantify the effects and accomplishments of its programs, which are consistent with OMB guidance. In order to carry out section 1105(a) (29) of GPRA, SAMHSA is required to prepare a performance plan for its major programs of activity. This plan must:
a) Establish performance goals to define the level of performance to be achieved by a program activity;
b) Express such goals in an objective, quantifiable, and measurable form;
c) Briefly describe the operational processes, skills and technology, and the human, capital, information, or other resources required to meet the performance goals;
d) Establish performance indicators to be used in measuring or assessing the relevant outputs, service levels, and outcomes of each program activity;
e) Provide a basis for comparing actual program results with the established performance goals; and
f) Describe the means to be used to verify and validate measured values.
In addition, this data collection supports the GPRA Modernization Act (GPRAMA) of 2010 which requires overall organization management to improve agency performance and achieve the mission and goals of the agency through the use of strategic and performance planning, measurement, analysis, regular assessment of progress, and use of performance information to improve the results achieved. Specifically, this data collection will allow CMHS to have the capacity to report on a consistent set of performance measures across its various grant programs that conduct each of these activities. SAMHSA’s legislative mandate is to increase access to high quality substance abuse and mental health prevention and treatment services and to improve outcomes. Its mission is to improve the quality and availability of treatment and prevention services for substance abuse and mental illness. To support this mission, the Agency’s overarching goals are:
Accountability—Establish systems to ensure program performance measurement and accountability
Capacity—Build, maintain, and enhance mental health and substance abuse infrastructure and capacity
Effectiveness—Enable all communities and providers to deliver effective services
Each of these key goals complements SAMHSA’s legislative mandate. All of SAMHSA’s programs and activities are geared toward the achievement of these goals and performance monitoring is a collaborative and cooperative aspect of this process.
SAMHSA will strive to coordinate the development of these goals with other ongoing performance measurement development activities. This information collection is needed to provide objective data to demonstrate SAMHSA’s monitoring and achievement of its mission and goals.
The urgency of health reform compelled SAMHSA to define recovery and to promote the availability, quality, and financing of vital services and supports that facilitate recovery for individuals. In addition, the integration mandate in Title II of the Americans with Disabilities Act and the Supreme Court’s decision in Olmstead v. L.C., 527 U.S. 581 (1999) provide legal requirements that are consistent with SAMHSA’s mission to promote a high-quality and satisfying life in the community for all Americans. In support of Strategic Initiative #4, SAMHSA first developed a working-definition of recovery which was arrived at via stakeholder input, a series of facilitated discussions, and a comprehensive review of the literature. These efforts resulted in the following definition of recovery: “A process of change through which individuals improve their health and wellness, live a self-directed life, and strive to reach their full potential.” After developing the recovery definition and supporting guiding principles, SAMHSA leadership sought a means for operationalizing recovery among the SAMHSA grantee population.
Individuals at three different levels will use the information: the SAMHSA Administrator and staff, the Center Directors and Project Officers, and grantees:
SAMHSA level—This information will be used to inform the administration on the performance of the programs funded through the Agency. Assessment of performance will be based on the new measures in line with the grant’s program goals as set by program leadership. The intent is that the information will serve as the basis of the annual performance report to Congress contained in the Justifications of Budget Estimates.
Center level—In addition to providing information on the performance of the various programs, the information can be used to monitor and manage individual grant projects within each program. The information can be used to identify strengths and weaknesses and provide an informed basis for providing technical assistance and other support to grantees, informing continuation funding decisions, and identifying potential subjects for further evaluation.
Grantee level—In addition to monitoring performance outcomes, the grantee staff can use the information to improve the quality of services that are provided to consumers within their projects, to promote service system capacity and infrastructure development, to prevent negative impacts of mental health problems, and to promote mental wellness.
To fulfill GPRA requirements, SAMHSA develops a report for each fiscal year that includes results of performance monitoring for the three preceding fiscal years. The additional information collected through this process will allow SAMHSA to report on the results of these performance outcomes as well as be consistent with the specific performance domains that SAMHSA is implementing to assess the accountability and performance of its discretionary and formula grant programs.
To facilitate SAMHSA-wide reporting, the agency has identified ten domains of particular interest for accountability and performance monitoring of client-level data for programs providing direct services. These domains are:
Access/Capacity
Functioning
Stability in Housing
Education and Employment
Crime and Criminal Justice
Perception of Care
Social Connectedness
Retention
Cost-Effectiveness
Evidence-Based Practices
As stated above, the SAMHSA CMHS programs that provide direct treatment to consumers, or Services programs, currently have an OMB approved data collection in place. Consequently, this request for approval of the adult services instrument is for a revision to the existing data collection instrument named the “CMHS NOMs Client-level Measures for Discretionary Programs Providing Direct Services Tool for Adult Programs.” The recovery measures will not be used in interviewing children (or their caregivers for children under the age of 11 years old). These SAMHSA TRAC data will be collected at baseline, at six month reassessments for as long as the consumer remains in treatment, and at discharge. The data collection encompasses eight of the ten SAMHSA NOMs domains.
Table 1. Data Collection for Client-level Measures
Domain |
Number of Questions: Adult |
Number of Questions: Child |
Access/Capacity |
4 |
4 |
Functioning |
28 |
26 |
Violence and Trauma |
6 |
0 |
Military Family Deployment |
6 |
6 |
Stability in Housing |
1 |
2 |
Education and Employment |
4 |
3 |
Crime and Criminal Justice |
1 |
1 |
Perception of Care |
15 |
14 |
Social Connectedness |
4 |
4 |
Retention |
5 |
5 |
Recovery |
12 |
0 |
Total Number |
87 |
65 |
To facilitate CMHS reporting of GPRA data for programs engaged in substantial infrastructure development, prevention, and mental health promotion activities, the agency has identified 14 categories of particular interest for accountability and performance monitoring. No changes are proposed for these categories. These categories are:
Policy Development
Workforce Development
Financing
Organizational Change
Partnerships/Collaborations
Accountability
Types/Targets of Practices
Awareness
Training
Knowledge/Attitudes/Beliefs
Screening
Outreach
Referral
Access
The following table summarizes the total number of indicators for each category that may or may not apply to each grant program:
Table 2. Data Collection for Infrastructure, Prevention, and Mental Health Promotion Indicators
Category |
Number of Indicators |
Policy Development |
2 |
Workforce Development |
5 |
Financing |
3 |
Organizational Change |
1 |
Partnerships/Collaborations |
2 |
Accountability |
6 |
Types/Targets of Practices |
4 |
Awareness |
1 |
Training |
1 |
Knowledge/Attitudes/Beliefs |
1 |
Screening |
1 |
Outreach |
2 |
Referral |
1 |
Access |
1 |
Total Number |
31 |
SAMHSA and CMHS intend to compare infrastructure, prevention, and mental health promotion targets set at baseline with data collected quarterly. These outcomes will be used as the indicator of performance.
Proposed Changes to Data Collection Tool
SAMHSA is proposing the revision of this data collection instrument in order to improve outcome measurement of its strategic initiative supporting recovery from mental health and substance use disorders. SAMHSA has developed a standard measure of recovery that can be used as part of its grantee performance reporting activities. SAMHSA’s Center for Behavioral Health Statistics and Quality (CBHSQ) previously conducted a pilot on its Recovery Measure. CMHS is intending to use part of the Recovery Measure as an element of its regular GPRA performance data collection.
In consultation with SAMHSA’s CMHS’ Recovery Measure Expert Panel, a group comprised of content experts from the recovery field as well as consumers in recovery themselves, the expert team developed a working definition of recovery and identified a need to develop an instrument that can be used to assess the degree to which recovery is occurring in SAMHSA funded programs. Staff from SAMHSA’s CBHSQ conducted an extensive literature review and developed an instrument consisting of 21 items to capture the four dimensions of recovery (i.e. home, health, purpose, and community) that contribute to SAMHSA’s working definition of recovery. Included in the CBHSQ instrument were eight items selected from the World Health Organization’s (WHO) Quality of Life (QOL) survey. CMHS incorporated twelve items from CBHSQ’s instrument into the CMHS NOMs Adult Client-level Measures for Discretionary Programs Providing Direct Services, including the 8 WHO QOL items and four additional questions that focused on substance abuse and mental health recovery and are of critical importance to measuring recovery within grantee populations .
The WHO QOL-8 will assess the following domains using the items listed below:
Question Number |
Item |
Domain |
1 |
How would you rate your quality of life? |
Overall quality of life |
2 |
How satisfied are you with your health? |
Overall quality of life |
3 |
Do you have enough energy for everyday life? |
Physical health |
4 |
How satisfied are you with your ability to perform your daily activities? |
Physical health |
5 |
How satisfied are you with yourself? |
Psychological |
6 |
How satisfied are you with your personal relationships? |
Social relationships |
7 |
Have you enough money to meet your needs? |
Environment |
8 |
How satisfied are you with the conditions of your living place? |
Environment |
In addition to the eight WHO QOL-8 items, the proposed Recovery Measure will include four existing measures used as part of the GPRA (Attachment 2). These outcome measures reflect SAMHSA’s desire for consistency in data collected within the Agency. SAMHSA has implemented specific performance domains called NOMs to assess the accountability and performance of its discretionary and formula grant programs. These domains represent SAMHSA’s focus on the factors that contribute to the success of mental health and substance abuse prevention and treatment. The Client/Participant Outcome Measures will address the following performance domains:
Abstinence from Drug / Alcohol Use
Employment / Education
Social Connectedness
Social Consequences from Drug / Alcohol Use
Access / Capacity
The revision also includes the following current recovery-related performance measures:
Question Number |
Item |
9 |
During the past 30 days, how much have you been bothered by these psychological or emotional problems? (This question will be placed in the instrument following the K6 questions for proper sequence). |
10 |
I have family or friends that are supportive of my recovery. |
11 |
I generally accomplish what I set out to do. |
12 |
I feel capable of managing my health care needs: |
SAMHSA will use the WHO QOL-8 items and GPRA measures to report on the performance of its discretionary services grant programs. The performance measures information is used by individuals at three different levels: the SAMHSA administrator and staff, the Center administrators and government project officers, and grantees:
These proposed data activities are intended to promote the use of consistent measures among CMHS-funded grantees and contractors. These measures are a result of extensive examination and recommendations, using consistent criteria, by panels of staff, experts, and grantees. Wherever feasible, the measures are consistent with or build upon previous data development efforts within CMHS. These data collection activities are organized to reflect and support the domains specified for SAMHSA’s NOMs for programs providing direct services, and the categories developed by CMHS to specify the infrastructure, prevention, and mental health promotion activities.
NO changes were made to the NOMs Client-Level Measures for Discretionary Programs Providing Direct Services Tool Child/Adolescent or Caregiver Combined Respondent Version and the Infrastructure Development, Prevention, and Mental Health Promotion Performance Indicators instruments.
Information technology will be used to reduce program respondent burden. The existing TRAC System is a web-based data entry and reporting system designed to support web-based data collection efforts for CMHS. The system will be updated to incorporate proposed changes to the client-level data collection and the infrastructure development, prevention, and mental health promotion performance indicators. One hundred percent of responses are expected to be submitted electronically through the web-based system. The TRAC System also provides a data repository service that includes methods for receiving the data, data quality checks, storage, and data presentation in reports by individual performance indicator or grouped with other performance indicators. The TRAC system complies with the requirements of Section 508 of the Rehabilitation Act to permit accessibility to people with disabilities.
This web-based system is intended to allow for easy data entry and access to reports for grantees that are required to submit TRAC data to CMHS. Entering and accessing data and viewing reports will be limited to those individuals with a username and password. A user’s level of access to the data and reports will be defined based on his or her authority and responsibilities.
Electronic submission of the data promotes enhanced data quality. With built-in data quality checks and easy access to data outputs and reports, users of the data can feel confident about the quality of the output. The electronic submission also promotes immediate access to the dataset. Once the data are put into the web-based system, it will be available for access, review, and reporting by all those with access to the system from Center staff to the grantee staff.
A program-level review of current measures and methods of collection was conducted to identify duplication of these data collection efforts. With the goal of creating standardized indicators and methods for monitoring grantee performance across the Center, existing measures were considered for use where appropriate. However, modification of current measures was necessary in some cases to generalize across varied programs. Each of these data collection instruments was reviewed and approved by the Government Project Officers, Branch Chiefs, and CMHS senior leadership as meeting the performance monitoring and management needs of individual programs and the Center. Since many of the grantees engaged in infrastructure development, prevention, and mental health promotion activities already collect data for the proposed indicators, the creation of this system will provide them with a standardized method for reporting to CMHS.
Individual grantees vary from small entities through large provider organizations. Every effort has been made to reduce the number of data items collected from grantees to the least number required to accomplish the objectives of the effort and to meet performance and GPRA reporting requirements and therefore, there is no significant impact involving small entities in general. Based on the pilot test and input and feedback from CMHS Project Officers, however, we understand that it may be difficult for some American Indian/Alaska Native Tribes and tribal organizations to report on the infrastructure development, prevention, and mental health promotion performance indicators on a quarterly basis. CMHS will, therefore, develop a waiver process to allow such grantees to request, through their Project Officers, to report on these indicators every six months rather than quarterly.
Client-level data
Mental health programs typically collect client-level data at admission and then conduct periodic reassessments of consumers while the individual remains in services. When feasible, mental health providers also conduct an assessment when the consumer is discharged. The data collection schedule for the client-level measures parallels this model. All programs that provide direct services will collect data every six months while the consumer is receiving services; this is a reduction from the prior requirement of quarterly data collection for three of the CMHS programs (the National Child Traumatic Stress Initiative, Meeting the Needs of Elderly Americans, and HIV/AIDS Minority Mental Health Services programs.)
The baseline data collection point is critical for measuring changes. Extending the interval for the periodic reassessment beyond the requested intervals could lead to loss of contact with consumers, significantly diminishing the response rates and lowering the value of the data for performance reporting use by losing measurement of intermediate effects.
Infrastructure development, prevention, and mental health promotion data
This quarterly data collection requirement for the infrastructure development, prevention, and mental health promotion performance indicators is necessary to provide CMHS with the information when needed for appropriate program monitoring and management, as well as for GPRA performance reporting.
This information collection fully complies with the guidelines in 5 CFR 1320.5(d) (2).
The notice required by 5 CFR 1320.8(d) was published in the Federal Register on October 7, 2016 (81 FR 69838). No comments were received in response to this notice.
Both external and internal stakeholders were consulted by CMHS in the development of these indicators and the data collection methodology. CMHS obtained feedback and consultation regarding the availability of data, methods and frequency of collection, and the appropriateness of data elements.
No monetary incentives are provided to grantees.
SAMHSA’s grantees do not collect individually identifiable information for these programs. Only aggregated data will be reported by grantees, therefore, SAMHSA and its contractors will not receive identifiable client records. Provider-level information will be aggregated to, at the least, the level of the grant/cooperative agreement-funding announcement.
No questions of a sensitive nature are asked of state, tribe, or U.S. territory grantees. However, grantees who are service providers collect information such as use of alcohol and other drugs and mental health conditions routinely as part of the provision of services. While some of this information may appear sensitive, program participants are service recipients.
The time to complete the instruments is estimated in Table 3. These estimates are based on current funding and planned fiscal year 2016 notice of funding announcements (NOFA), SAMHSA CMHS discretionary programs that will use these measures in fiscal years 2017 through 2019 to collect client-level data, and grantee reports of the amount of time required to complete the currently approved instruments accounting for the additional time required to complete the new questions, as based on an informal pilot and prior CMHS experience in collecting similar data.
Table 3. Estimates of Annualized Hour Burden
Type of Response |
Number of Respondents |
Responses per Respondent |
Total Responses |
Hours per Response |
Total Hour Burden |
Hourly Wage Cost |
Total Hour Cost |
Client-level baseline interview |
55,744 |
1 |
55,744 |
0.58 |
32,332 |
$20.641 |
$667,323 |
Client-level 6-month reassessment interview2 |
44,595 |
1 |
44,595 |
0.58 |
25,865 |
$20.64 |
$533,856 |
Client-level discharge interview3 |
16,723 |
1 |
16,723 |
0.58 |
9,699 |
$20.64 |
$200,194 |
PBHCI- Section H Form Only Baseline |
14,000 |
1 |
14,000 |
.08 |
1,120 |
$20.64 |
$23,117 |
PBHCI- Section H Form Only Follow-Up4 |
9,240 |
1 |
9,240 |
.08 |
739 |
$20.64 |
$15,257 |
PBHCI – Section H Form Only Discharge5 |
4,200 |
1 |
4,200 |
.08 |
336 |
$20.64 |
$6,935 |
HIV Continuum of Care Specific Form Baseline |
200 |
1 |
200 |
0.33 |
66 |
$20.64 |
$1,362 |
HIV Continuum of Care Follow-Up6 |
148 |
1 |
148 |
0.33 |
49 |
$20.64 |
$1,008 |
HIV Continuum of Care Discharge7 |
104 |
1 |
104 |
0.33 |
34 |
20.64 |
$708 |
Subtotal |
144,954 |
|
144,954 |
|
70,240 |
|
$1,449,760 |
Infrastructure development, prevention, and mental health promotion quarterly record abstraction8 |
982 |
4.0 |
3,928 |
2.0 |
7,856 |
$20.64 |
$162,148 |
Total |
145,936 |
|
148,882 |
|
78,096 |
|
$1,611,908 |
There will be no capital, start-up, operation, maintenance, nor purchase costs incurred by the mental health programs participating in this CMHS data collection, or by consumers receiving CMHS-funded services.
The principal additional cost to the government for this project is the cost of a contract to collect the data from the various programs and to conduct analyses which generate routine reports from the data collected. The reports examine baseline characteristics as well as the changes between baseline, discharge, and each of the follow-up periods. It is the responsibility of the contractor to work with the Government Project Officer (GPO) when preparing reports that combine the client services data with the annual reports of the project.
The estimated annualized cost for a contract for the GPRA mandate is $1,859,161 and the cost of 1 FTE staff (GS-14 100%) responsible for the CMHS data collection effort is approximately $132,000 /year. The estimated annualized total cost to the government will be $1,991,161.
Currently, there are 48,815 total burden hours in the OMB inventory and CMHS is requesting 78,096 burden hours. The increase of 29,281 is a program change in burden due to the number of overall grant programs and number of respondents receiving mental health treatment services.
SAMHSA/CMHS utilizes the NOMs data on an ongoing basis to monitor performance and to respond to GPRAMA and other Federal reporting requirements. These data are used to provide the agency with information to document the overall Center performance requirements and to provide information that will assist CMHS in planning and monitoring program goals. Descriptive information obtained from program reporting requirements will be reviewed for monitoring and program management. Information is used internally by the agency and for performance reports. There are no formal publication plans.
The time frame for submission of the reporting requirements varies by grant cycle and grant program period of performance throughout the year.
The expiration date for OMB approval will be displayed on all data collection instruments.
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions. The certifications are included in this submission.
1 The hourly wage estimate is $20.13 based on the Occupational Employment and Wages, May 2014 Mean Hourly Wage Rate for 21-1011 Substance Abuse and Behavioral Disorder Counselors = $20.13/hr. as of May, 2014. (http://www.bls.gov/oes/current/oes211011.htm (Accessed on August 24, 2015).
2 It is estimated that 66% of baseline clients will complete this interview.
3 It is estimated that 30% of baseline clients will complete this interview.
4 It is estimated that 74% of baseline clients will complete this interview.
5 It is estimated that 52% of baseline clients will complete this interview.
6 It is estimated that 52% of baseline clients will complete this interview.
7 It is estimated that 30% of baseline clients will complete this interview.
8 Grantees are required to report this information as a condition of their grant. No attrition is estimated.
Note: Numbers may not add to the totals due to rounding and some individual participants completing more than one form.
| File Type | application/msword |
| File Title | Transformation Accountability (TRAC) Reporting System |
| Author | Jessica Taylor |
| Last Modified By | Windows User |
| File Modified | 2016-12-12 |
| File Created | 2016-09-07 |