Transformation Accountability (TRAC) Reporting System
Supporting Statement
The Substance Abuse and Mental Health Services Administration’s (SAMHSA), Center for Mental Health Services (CMHS) is requesting approval from the Office of Management and Budget (OMB) for a revision to the TRAC reporting system data collection which include the following three instruments:
1. The CMHS NOMs Adult Client-level Measures for Discretionary Programs Providing Direct Services (Attachment 1);
2. The CMHS NOMs Child Client-level Measures for Discretionary Programs Providing Direct Services (Child/Caregiver Version) (Attachment 2); and
3. The Infrastructure Development, Prevention, and Mental Health Promotion Performance Indicators (Attachment 3).
The first two instruments are revisions to the National Outcome Measures (NOMs) for Consumers Receiving Mental Health Services (OMB No. 0930-0285, Expiration Date: 4/30/2010); a currently OMB-approved data collection effort directed at consumers of the Services (NOMs) programs. Additional questions were added to these instruments to enable CMHS to more fully explain grantee performance in relation to Agency and/or program objectives. The third instrument is a new survey directed at the Grant Project Directors of the Infrastructure, Prevention, and Mental Health Promotion grant programs. This new instrument will enable SAMHSA CMHS to capture a standardized set of performance indicators using a uniform reporting method.
This information collection will allow SAMHSA to continue to meet the Government Performance and Results Act of 1993 (GPRA) reporting requirements that quantify the effects and accomplishments of its programs, which are consistent with OMB guidance. In order to carry out section 1105(a) (29) of GPRA, SAMHSA is required to prepare a performance plan for its major programs of activity. This plan must:
Establish performance goals to define the level of performance to be achieved by a program activity;
Express such goals in an objective, quantifiable, and measurable form;
Briefly describe the operational processes, skills and technology, and the human, capital, information, or other resources required to meet the performance goals;
Establish performance indicators to be used in measuring or assessing the relevant outputs, service levels, and outcomes of each program activity;
Provide a basis for comparing actual program results with the established performance goals; and
Describe the means to be used to verify and validate measured values.
This data collection will allow CMHS to have the capacity to report on a consistent set of performance measures across its various grant programs that conduct each of these activities. SAMHSA’s legislative mandate is to increase access to high quality substance abuse and mental health prevention and treatment services and to improve outcomes. Its mission is to improve the quality and availability of treatment and prevention services for substance abuse and mental illness. To support this mission, the Agency’s overarching goals are:
Accountability—Establish systems to ensure program performance measurement and accountability
Capacity—Build, maintain, and enhance mental health and substance abuse infrastructure and capacity
Effectiveness—Enable all communities and providers to deliver effective services
Each of these key goals complements SAMHSA’s legislative mandate. All of SAMHSA’s programs and activities are geared toward the achievement of these goals and performance monitoring is a collaborative and cooperative aspect of this process.
SAMHSA will strive to coordinate the development of these goals with other ongoing performance measurement development activities. This information collection is needed to provide objective data to demonstrate SAMHSA’s monitoring and achievement of its mission and goals.
These proposed data activities are intended to promote the use of consistent measures among CMHS-funded grantees and contractors. These common measures recommended by CMHS are a result of extensive examination and recommendations, using consistent criteria, by panels of staff, experts, and grantees. Wherever feasible, the measures are consistent with or build upon previous data development efforts within CMHS. These data collection activities will be organized to reflect and support the domains specified for SAMHSA’s NOMs for programs providing direct services, and the categories developed by CMHS to specify the infrastructure, prevention, and mental health promotion activities.
Individuals at three different levels will use the information: the SAMHSA Administrator and staff, the Center Directors and Project Officers, and grantees:
SAMHSA level—This information will be used to inform the administration on the performance of the programs funded through the Agency. Assessment of performance will be based on the new measures in line with the grant’s program goals as set by program leadership. The intent is that the information will serve as the basis of the annual performance report to Congress contained in the Justifications of Budget Estimates.
Center level—In addition to providing information on the performance of the various programs, the information can be used to monitor and manage individual grant projects within each program. The information can be used to identify strengths and weaknesses and provide an informed basis for providing technical assistance and other support to grantees, informing continuation funding decisions, and identifying potential subjects for further evaluation.
Grantee level—In addition to monitoring performance outcomes, the grantee staff can use the information to improve the quality of services that are provided to consumers within their projects, to promote service system capacity and infrastructure development, to prevent negative impacts of mental health problems, and to promote mental wellness.
To fulfill GPRA requirements SAMHSA develops a report for each fiscal year that includes results of performance monitoring for the three preceding fiscal years. The additional information collected through this process will allow SAMHSA to report on the results of these performance outcomes as well as be consistent with the specific performance domains that SAMHSA is implementing to assess the accountability and performance of its discretionary and formula grant programs.
To facilitate SAMHSA-wide reporting, the agency has identified ten domains of particular interest for accountability and performance monitoring of client-level data for programs providing direct services. These domains are:
Access/Capacity
Functioning
Stability in Housing
Education and Employment
Crime and Criminal Justice
Perception of Care
Social Connectedness
Retention
Cost-Effectiveness
Evidence-Based Practices
As stated above, the SAMHSA CMHS programs that provide direct treatment to consumers, or Services programs, currently have an OMB-approved data collection in place. Consequently, this request for approval of the two Services instruments is for revisions to the existing data collection instruments. This data collection includes separate data collection forms that are parallel in design for use in interviewing adults and children (or their caregivers for children under the age of 11 years old); named the CMHS NOMs Adult Client-level Measures for Discretionary Programs Providing Direct Services and the Child Client-level Measures for Discretionary Programs Providing Direct Services, respectively. These SAMHSA TRAC data will be collected at baseline, at six month reassessments for as long as the consumer remains in treatment, and at discharge. The data collection encompasses eight of the ten SAMHSA NOMs domains.
Table 1. Data Collection for Client-level Measures
Domain |
Number of Questions: Adult |
Number of Questions: Child |
Access/Capacity |
4 |
4 |
Functioning |
28 |
26 |
Stability in Housing |
1 |
2 |
Education and Employment |
4 |
3 |
Crime and Criminal Justice |
1 |
1 |
Perception of Care |
15 |
14 |
Social Connectedness |
4 |
4 |
Retention |
5 |
5 |
Total Number |
63 |
59 |
The current OMB-approved instruments were revised as follows:
The names of the instruments were changed to simplify and establish a single clearer terminology for referring to the instruments that is more acceptable to diverse grantees.
The administrative section of all instruments was changed to allow grantees to capture and track when consumers refuse interviews, consent cannot be obtained from proxy, and consumers are impaired or unable to provide consent. The administrative section of the children’s instruments was additionally changed to capture whether the respondent is the child or his/her caregiver.
Questions were added to all instruments to capture general health, psychological functioning, life in the community, and substance use.
CMHS reduced the data collection requirement for 3-month programs to be consistent with 6-month programs; all grant programs will be required to collect the client-level data in 6 month intervals. CMHS will require the collection of Clinical Discharge interviews.
In addition to questions asked of consumers as listed above, programs will be required to abstract information from consumer records regarding the services provided at reassessment and discharge. The revised instruments are included as Attachment 1—Adult Client-level Measures and Attachment 2—Child Client-level Measures.
SAMHSA and CMHS intend to compare client-level data collected at baseline with periodic reassessments. These outcomes will be used as the indicator of performance.
To facilitate CMHS reporting of GPRA data for programs engaged in substantial infrastructure development, prevention, and mental health promotion activities, the agency has identified 14 categories of particular interest for accountability and performance monitoring. These categories are:
Policy Development
Workforce Development
Financing
Organizational Change
Partnerships/Collaborations
Accountability
Types/Targets of Practices
Awareness
Training
Knowledge/Attitudes/Beliefs
Screening
Outreach
Referral
Access
CMHS has identified associated grant-level indicators, to be reported by the grant Project Directors, to assess performance within these categories. These performance indicators are the focus of the third data collection instrument: Attachment 3—Infrastructure Development, Prevention, and Mental Health Promotion Performance Indicators. A web-based data entry system will be developed to capture this performance data for all CMHS-funded Infrastructure, Prevention, and Mental Health Promotion grants.
Not all categories or indicators will apply to every grant program; CMHS program leaders will be responsible for determining whether a category (or an indicator within a category) applies to each grant program, establishing targets at the individual grant level. Project Officers will be responsible for monitoring and reviewing the data submitted. Grantee Project Directors will be responsible for submitting data quarterly. While many programs are currently collecting these data, the use of standardized categories and data collection approaches will enhance aggregate data development and reporting.
The following table summarizes the total number of indicators for each category that may or may not apply to each grant program:
Table 2. Data Collection for Infrastructure, Prevention, and Mental Health Promotion Indicators
Category |
Number of Indicators |
Policy Development |
2 |
Workforce Development |
5 |
Financing |
3 |
Organizational Change |
1 |
Partnerships/Collaborations |
2 |
Accountability |
6 |
Types/Targets of Practices |
4 |
Awareness |
1 |
Training |
1 |
Knowledge/Attitudes/Beliefs |
1 |
Screening |
1 |
Outreach |
2 |
Referral |
1 |
Access |
1 |
Total Number |
31 |
SAMHSA and CMHS intend to compare infrastructure, prevention, and mental health promotion targets set at baseline with data collected quarterly. These outcomes will be used as the indicator of performance.
Information technology will be used to reduce program respondent burden. The existing TRAC System is a web-based data entry and reporting system designed to support web-based data collection efforts for CMHS. The system will be updated to incorporate proposed changes to the client-level data collection and the infrastructure development, prevention, and mental health promotion performance indicators. 100% of responses are expected to be submitted electronically through the web-based system. The TRAC System also provides a data repository service that includes methods for receiving the data, data quality checks, storage, and data presentation in reports by individual performance indicator or grouped with other performance indicators. The TRAC system complies with the requirements of Section 508 of the Rehabilitation Act to permit accessibility to people with disabilities.
This web-based system is intended to allow for easy data entry and access to reports for grantees that are required to submit TRAC data to CMHS. Entering and accessing data and viewing reports will be limited to those individuals with a username and password. A user’s level of access to the data and reports will be defined based on his or her authority and responsibilities.
Electronic submission of the data promotes enhanced data quality. With built-in data quality checks and easy access to data outputs and reports, users of the data can feel confident about the quality of the output. The electronic submission also promotes immediate access to the dataset. Once the data are put into the web-based system, it will be available for access, review, and reporting by all those with access to the system from Center staff to the grantee staff. All data entry screens will include the OMB approval information (see Attachment 4—Screen Shots).
This data collection is necessary in order to assess grantee performance. Although individual CMHS programs that conduct infrastructure development or prevention, and/or mental health promotion activities currently collect and report some type of performance data, CMHS does not currently have standard reporting for these types of activities. Instead, individual grant programs are independently collecting data on a variety of indicators using different types of measures.
A program-level review of current measures and methods of collection was conducted to identify duplication of these data collection efforts. With the goal of creating standardized indicators and methods for monitoring grantee performance across the Center, existing measures were considered for use where appropriate. However, modification of current measures was necessary in some cases to generalize across varied programs. Each of these data collection instruments was reviewed and approved by the Government Project Officers, Branch Chiefs, and CMHS senior leadership as meeting the performance monitoring and management needs of individual programs and the Center. Since many of the grantees engaged in infrastructure development, prevention, and mental health promotion activities already collect data for the proposed indicators, the creation of this system will provide them with a standardized method for reporting to CMHS.
Individual grantees vary from small entities through large provider organizations. Every effort has been made to reduce the number of data items collected from grantees to the least number required to accomplish the objectives of the effort and to meet performance and GPRA reporting requirements and therefore, there is no significant impact involving small entities in general. Based on the pilot test and input and feedback from CMHS Project Officers, however, we understand that it may be difficult for some American Indian/Alaska Native Tribes and tribal organizations to report on the infrastructure development, prevention, and mental health promotion performance indicators on a quarterly basis. We will, therefore, develop a waiver process to allow such grantees to request, through their Project Officers, to report on these indicators every six months rather than quarterly.
Client-level data
Mental health programs typically collect client-level data at admission and then conduct periodic reassessments of consumers while the individual remains in services. When feasible, mental health providers also conduct an assessment when the consumer is discharged. The data collection schedule for the client-level measures parallels this model. All programs that provide direct services will collect data every six months while the consumer is receiving services; this is a reduction from the prior requirement of quarterly data collection for three of the CMHS programs (the National Child Traumatic Stress Initiative, Meeting the Needs of Elderly Americans, and HIV/AIDS Minority Mental Health Services programs.)
The baseline data collection point is critical for measuring changes. Extending the interval for the periodic reassessment beyond the requested intervals could lead to loss of contact with consumers, significantly diminishing the response rates and lowering the value of the data for performance reporting use by losing measurement of intermediate effects.
Infrastructure development, prevention, and mental health promotion data
This quarterly data collection requirement for the infrastructure development, prevention, and mental health promotion performance indicators is necessary to provide CMHS with the information when needed for appropriate program monitoring and management, as well as for GPRA performance reporting.
This information collection fully complies with the guidelines in 5 CFR 1320.5(d) (2).
The notice required in 5 CFR 1320.8(d) was published in the Federal Register on October 6, 2009 (74 FR 51280). One set of comments was received from a grantee currently reporting data using the previously approved client-level instrument (Attachment 5—Public Comments in Response to the Federal Register Notice). The following paragraphs summarize these comments and provide SAMHSA’s response.
Comment 1: The respondent time for completing the previously approved client-level instrument is substantially longer for Latino, monolingual Spanish-speaking older adults with both physical and mental health symptoms who are first generation immigrants. Consequently, the additional items proposed represent a significant burden.
SAMHSA’s Response to Comment 1: The time to complete, as reported on the data collection instruments, is an average and subsequently may vary for individuals based on individual characteristics. In response to this comment, CMHS conducted an informal random poll of nine grantees that currently collect TRAC client-level data to better understand the extent of this issue. The results indicated that the time to complete the existing instrument is approximately 18 minutes on average. CMHS recognizes the multiple challenges faced by this unique population and will provide technical assistance to identify possible solutions for minimizing the additional burden the revisions represent. CMHS will monitor the progress of this grantee and work with them accordingly to reduce the burden associated with these data collection efforts.
Comment 2: “Despite the existence of an approved Spanish translation of the CMHS-TRAC NOMs tool,” the lack of equivalent concepts poses an additional challenge in completing client-level interviews. Specific comments included issues regarding: race (“program participants think about their heritage in terms of their ethnicity and country of origin”); the reversed scoring of a few questions; and substance use terminology.
SAMHSA’s Response to Comment 2: CMHS is sensitive to addressing the needs of various cultural groups in regards to concepts, language, and customs in the process of developing data collection instruments used by CMHS grantees. With regard to the question of how race is worded, CMHS used the categories published by OMB in the Federal Register on October 30, 1997. The Center for Substance Abuse Treatment within SAMHSA uses the same wording for collecting race data, thereby allowing SAMHSA to aggregate data across Centers.
Regarding the issue of reversed scoring, whenever possible, CMHS used validated instruments from which to draw questions. The comments are in reference to items drawn from the CMHS Mental Health Statistics Improvement Program (MHSIP) survey that is used by the CMHS-administered Community Mental Health Services Block Grant Program. Moreover, including reversed scored questions is a known method for reducing the effect of response biases.
The substance use terminology issue also is addressed by the use of validated instruments; these questions were drawn from the World Health Organization’s (WHO) Alcohol, Smoking and Substance Involvement Screening Test (ASSIST). The instrument will be translated into Spanish using a translation – back translation method. Despite potential difficulties expressed by this grantee, CMHS encourages the attempt to collect these important data as SAMHSA is making efforts to reduce the high incidence of co-occurring substance use and mental health disorders. Technical assistance will be provided to grantees that may have difficulty in collecting data in response to this new set of questions.
Both external and internal stakeholders were consulted by CMHS in the development of these indicators and the data collection methodology. CMHS obtained feedback and consultation regarding the availability of data, methods and frequency of collection, and the appropriateness of data elements.
Program |
Grant Organization Name |
Contact Name |
Phone |
Earmark Adults Grant |
United Community Center |
Cindy Kay Suszek |
(414) 643-8530 x1922 |
Earmark Adults Grant |
Marion County Health Department |
Scott Richards |
(503) 361-2695
|
Jail Diversion |
Department of Mental Health & Addiction Services (DMHAS) |
Linda Frisman |
(860) 418-6788 |
Older Adults |
Advocates, Inc. |
Lynn Kerner |
(508) 661-2028 |
Older Adults |
United Community Center |
Cindy Kay Suszek |
(414) 643-8530 x1922 |
HIV/AIDS |
Wayne State University |
Lori Zeman |
(313) 577-8172 |
HIV/AIDS |
Emory University |
Eugene Farber |
(404) 616-6862 |
SSH |
Southern California Alcohol & Drug Programs |
Keke Williams |
(562) 923-4545 x2243 |
SSH |
Pine Street Inn |
Lynne D. Chapman |
(617) 782-9201 |
Development of the new infrastructure development, prevention, and mental health promotion performance indicators involved extensive consultation with staff within CMHS and SAMHSA; in many instances, CMHS staff also sought feedback from their grantees to inform their thinking. The process began in 2004 with the formation of an internal CMHS Steering Committee with representatives of all CMHS Branches to oversee the development of TRAC. A subcommittee of the Steering Committee was formed to focus specifically on the development of infrastructure development performance indicators. The subcommittee began their process of identifying indicators by reviewing all of the infrastructure development performance indicators that individual programs were collecting at the time, many of which were developed with extensive input from the grantees in the particular program. Simultaneously during 2004, CMHS had developed broad categories of indicators for the evaluation of one large infrastructure development program, the Mental Health Transformation State Incentive Grant (MHT SIG) program (OMB No. 0930-0292), which the subcommittee decided to adopt for use across CMHS programs. Specific indicators that might apply across diverse CMHS programs were developed for each of the categories, based on the types of information programs were already collecting. The proposed indicators were presented to the Steering Committee for approval and to all CMHS staff at an All-Hands meeting in March of 2005. In December 2006, a two-day workshop was held to discuss potential methods for the standardized measurement and data collection for these indicators. The information and ideas discussed at this meeting were incorporated into the current methods and measures. Individuals from outside the agency who participated in the December 2006 meeting included:
Agency/Organization |
Contact Name |
Phone |
Macro International, Atlanta, GA |
John Gilford |
(404) 321-3211 |
Social Science Research and Evaluation, Inc. Burlington, MA |
Wayne Harding |
(781) 270-6613 |
Human Services Research Institute, Cambridge, MA |
H. Stephen Leff |
(617) 947-2148 |
University of Colorado at Denver / American Indian and Alaska Native Programs |
Doug Novins |
(303) 724-1414 |
After the December 2006 meeting, further development of the infrastructure development, prevention, and mental health promotion indicators was put on hold while the client-level measures were further developed. During the interim, the MHT SIG program began collecting data pertaining to the infrastructure development indicators. In the spring of 2009, CMHS renewed its concentration on the infrastructure development, prevention, and mental health promotion indicators through a new TRAC management team that includes representatives of programs, evaluators, and managers across CMHS. The new team re-examined the previously proposed indicators in light of data being collected by current grant programs. After adjusting the indicators in response to lessons learned from MHT SIG and other current programs, the TRAC management presented the proposed indicators to all CMHS staff through an all-hands meeting and internal SharePoint website and asked for feedback during a month-long period; some CMHS Project Officers shared the proposed indicators with their grantees to incorporate their feedback.
The final set of indicators was pilot-tested among a select number of grantees in November 2009. Their feedback primarily focused on topics affecting training and did not warrant change to the indicators themselves. The following grantees participated in the pilot test for these indicators:
Program |
Grant Organization Name |
Contact Name |
Phone |
Circles of Care |
American Indian Center of Chicago |
Amy West |
(312) 996-1077 |
Circles of Care |
Pueblo of San Felipe |
Deborah Altschul |
(505) 272-1786 |
Jail Diversion |
CT Department of Mental Health and Addiction services |
Jim Tackett |
(860) 418-6979 |
HIV |
Wayne State University |
Lori Zeman |
(313) 577-8172 |
GLS Campus |
Ohio State |
Winger Wendy
|
(614) 688-5829 |
SSH |
Community Connections |
Richard Bebout |
(202) 546-1512 |
SSH |
St. Vincent De Paul |
Julie Dede |
(619) 233-1060 |
Older Adults |
Mid Kansas Senior Outreach – Mental Health Association of South Central Kansas |
Don Strong |
(316) 685-1821 x235 |
Older Adults |
OASIS – Oakland Family Services, Pontiac Michigan |
Micheline Sommers |
(248) 858-7766 x246 |
No monetary payment will be made to the mental health programs or to the consumers participating in data collection. No monetary payment is directly paid to grantees for the submission of the GPRA data. They are expected to provide this information as a requirement of their grant award.
The client-level measures require grantees to interview all consumers that they serve. The use of incentives is addressed in each individual CMHS program’s Request for Application, which provides guidance on whether incentives are or not allowed. Preferred incentives include food vouchers, transportation vouchers, or phone cards.
For the client-level data collection process, program respondents will be expected to meet the requirements of the HIPAA and its associated Privacy Rule that sets the standards for the use and disclosure of an individual’s health/mental health information. Since the data reported for each consumer will be provided to the CMHS contractor only by number and not by name, the data cannot be directly linked to a specific person. The grantee providing the data will maintain the link between the identifier and the name of the consumer. The CMHS contractor will not have access to existing consumer clinical records, which are under the control of the respondent programs. Neither the CMHS contractor nor CMHS can link individual consumers to the data reported by the respondent programs.
The infrastructure development, prevention, and mental health promotion data collection processes do not involve gathering client-level information. Program respondents will be expected to meet the requirements of the HIPAA and its associated Privacy Rule that sets the standards for the use and disclosure of an individual’s health/mental health information.
This project was given an exemption by Westat’s Internal Review Board (IRB) because it is considered performance reporting and no individual consumer identifiers are collected or submitted to Westat (Attachment 6—IRB Exemption).
SAMHSA’s mission is to improve the quality and availability of prevention, early intervention, treatment, and rehabilitation services for substance abuse and mental illnesses, including co-occurring disorders, in order to improve health and reduce illness, death, disability, and cost to society. In carrying out this mission it is necessary for grantees providing direct services to collect sensitive items such as criminal justice involvement, substance use, and data related to mental health functioning. The data that will be submitted by each grantee will be based in large part on data that most of the programs are already routinely collecting. This primarily includes data on consumer demographics, mental health condition/illness and treatment history, services received, and consumer outcomes. These issues are essential to the service context. Many grant projects use informed consent forms as required and as viewed appropriate by their individual organizations. They use the appropriate forms for minor/adolescent participants requiring parental approval.
The time to complete the revised instruments is estimated in Table 3. These estimates are based on grantee reports of the amount of time required to complete the currently approved instruments accounting for the additional time required to complete the new questions, as based on an informal pilot and prior CMHS experience in collecting similar data.
Table 3. Estimates of Annualized Hour Burden
Type of Response |
Number of Respondents |
Responses per Respondent |
Total Responses |
Hours per Response |
Total Hour Burden |
Hourly Wage Cost |
Total Hour Cost |
Client-level baseline interview |
15,681 |
1 |
15,681 |
0.333 |
5,222 |
$151 |
$78,330 |
Client-level 6-month reassessment interview |
10,646 |
1 |
10,646 |
0.367 |
3,907 |
$15 |
$58,605 |
Client-level discharge interview2 |
4,508 |
1 |
4,508 |
0.367 |
1,655 |
$15 |
$24,825 |
Client-level baseline chart abstraction |
2,352 |
1 |
2,352 |
0.1 |
235 |
$15 |
$3,525 |
Client-level reassessment chart abstraction3 |
9,017 |
1 |
9,017 |
0.1 |
902 |
$15 |
$13,530 |
Client-level Subtotal4 |
15,681 |
|
15,681 |
|
11,921 |
$15 |
$178,815 |
Infrastructure development, prevention, and mental health promotion quarterly record abstraction |
942 |
4 |
3,768 |
4 |
15,072 |
$355 |
$527,520 |
TOTAL |
16,623 |
|
|
|
26,993 |
|
$706,335 |
There will be no capital, start-up, operation, maintenance, nor purchase costs incurred by the mental health programs participating in this CMHS data collection, or by consumers receiving CMHS-funded services.
The total contract award to cover all aspects of the design of the study, sampling design, data collection, and development of the data files, data tapes, and technical documentation is $8,807,858 over a 36-month period. Thus, the annualized contract cost is $2,935,953.
Additional costs will be incurred indirectly by the government in personnel costs of staff involved in oversight of data collection. It is estimated that one CMHS employee will each be involved for 100 percent of their time. Costs of CMHS staff time will approximate $120,000 annually.
The estimated annualized total cost to the government will be $2,989,494.
Currently there are 14,293 burden hours in the OMB inventory. CMHS is now requesting 26,992 hours. This increase of 12,699 hours is due to a program change.
Extra time is being allocated to the grant programs providing direct services to complete the revised client-level instruments (Attachments 1 & 2); it was estimated the additional questions would require, on average, an extra 10 minutes per interview per respondent.
The new instrument for the infrastructure development, prevention, and mental health promotion data collection was piloted with nine grantees. Based on the reported average time to complete, is estimated that on average the data collection and reporting for this effort will require an additional 16 hours per respondent per year.
Data for the annual performance plan/report are needed by SAMHSA on an ongoing basis. Data collection will commence with approval from OMB. Data are provided by CMHS for the most recently completed calendar year to SAMHSA each May in order to assure analysis in time for the annual performance report. The annual performance report must be submitted to the Department of Health and Human Services and to OMB by September and is included in the President's Annual Budget Request, which is released to the public February 1st. Data may be refined and added to the final Presidential Budget Request after the Department submits its initial performance report.
Because the expiration date for the previously approved client-level measures is April 30, 2010, data collection must begin by May 1, 2010 in order to avoid gaps in ongoing data collection.
Data will be available to CMHS staff and grantees through a series of reports available through the web-based TRAC system. Assigned roles determine user access. Individual grantees will only be allowed detailed access to data from their grant. They will also have access to summary information across all grantees in their program. CMHS staff access will be determined by their span of responsibility.
The TRAC System includes web-based reports of the current client-level data including information on the number of consumers served, their demographic characteristics, baseline status, and change scores for the various outcome domains. These data and the additional items will be analyzed and presented in performance reports using basic descriptive statistics. On the principle outcome items (i.e., the 8 NOMs domains covered), a comparison of consumer status after receiving services with baseline data will be used to assess any change in status; users will also be able to compare any of the interviews completed by a consumer. The web-based reports will also allow users to create basic cross tabulations of the data.
Web-based reports will be built for the infrastructure development, prevention, and mental health promotion data collection efforts incorporating information related to the categories and indicators described above.
Data will be used to report to Congress regarding the CMHS’ performance as specified in the SAMHSA Annual Justifications of Budget Estimates. This will also allow CMHS staff to examine performance longitudinally, by program, or individual grantee.
In addition to the reports on the TRAC website, data will be utilized for specialized analyses as needs emerge. Individual grantees will be able to download their own data into an Excel spreadsheet for further manipulation or to transfer to a statistical package.
The expectation is that over time the results will be examined for subpopulations of interest within individual activities or in response to emerging policy issues. With these analyses the data would be exported to a statistical package such as SAS for more elaborate analytic work.
The expiration date for OMB approval will be displayed on all data collection instruments (see Attachment 4—Screen Shots).
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions. The certifications are included in this submission.
All CMHS-funded discretionary grantees that provide direct services or engage in infrastructure development, prevention, or mental health promotion activities are required to participate in this data collection effort and will submit data based on their conducted activities. The table below indicates the number of grant programs (with the number of active grantees in FY 2010) for each of the TRAC data collection efforts:
Table 4. Data Collection Effort by CMHS-funded Program and Number of Active Grantees in FY 2010
CMHS-funded Program |
Total Number of Grants |
Client-level |
Infrastructure Development, Prevention, and Mental Health Promotion |
Child and Adolescent Mental Health and Substance Abuse State Infrastructure Grants |
7 |
No |
Yes |
Circles of Care |
8 |
No |
Yes |
Comprehensive Community Mental Health Services for Children and their Families Program |
65 |
Yes |
Yes |
Earmarks |
37 |
Yes |
Yes |
Garrett Lee Smith Campus Suicide Prevention Grant Program |
40 |
No |
Yes |
Garrett Lee Smith State/Tribal Suicide Prevention Grant Program |
49 |
No |
Yes |
Healthy Transitions Initiative |
7 |
Yes |
Yes |
Historically Black Colleges & Universities National Resource Center |
1, with 20 sub-grantees |
No |
Yes |
Jail Diversion |
14 |
Yes |
Yes |
Linking Actions for Unmet Needs in Children’s Mental Health |
18 |
No |
Yes |
National Suicide Prevention Lifeline |
1 |
No |
Yes |
Native Aspirations |
1 |
No |
Yes |
NCTSI Treatment & Service Centers |
13 |
Yes |
Yes |
NCTSI Community Treatment Centers |
44 |
Yes |
Yes |
NCTSI National Coordinating Center |
1 |
No |
Yes |
Mental Health Transformation State Incentive Grant |
9 |
Yes |
Yes |
Minority AIDS/HIV Services Collaborative Program |
16 |
Yes |
No |
Minority Fellowship Program |
5 |
No |
Yes |
Older Adults Targeted Capacity Expansion |
10 |
Yes |
No |
Primary and Behavioral Health Care Integration |
11 |
Yes |
Yes |
Safe Schools/Health Students Initiative |
175 |
No |
Yes |
Services in Supportive Housing |
55 |
Yes |
No |
State Mental Health Data Infrastructure Grants for Quality Improvement |
54 |
No |
Yes |
Statewide Consumer Network Grants |
31 |
No |
Yes |
Statewide Family Networks Grants |
48 |
No |
Yes |
Suicide Lifeline Crisis Center FUP Grants |
6 |
No |
Yes |
Total |
726 |
281 |
664 |
Information data collection procedures will be the responsibility of individual grantees and may vary by type of program.
Some grantees have service providers conduct client-level baseline and follow-up assessments, while others have grant evaluators perform this function.
Some grantees may wish to collect client-level information using paper and pencil methods. CMHS will provide downloadable paper versions of the data collection instruments to facilitate this process. These grantees will then submit their data electronically via a web-based data entry process. The data for those consumers with both baseline and periodic reassessment data are matched using a unique encrypted consumer identifier developed by the grantee. Grantees will be clearly instructed not to use identifying information (i.e., social security number or initials) as the consumer identifier.
Required data collection points are:
BASELINE: For consumers who have not previously been seen by the grantee, baseline data will be collected at admission. For consumers already enrolled in the program and continuing to receive services, administrative data should be submitted by the grantee within 30 days of initiating TRAC data collection. The timing of any subsequent data collection point(s) will be anchored to the baseline point the grantee indicates in this administrative record.
REASSESSMENT: CMHS requires client-level data collection every six months while the consumer is receiving CMHS-funded services. Ongoing periodic status review is viewed as consistent with good clinical practice.
DISCHARGE: Grantees must provide information on the type of discharge on all consumers who are discharged. When the discharge is a planned event, the consumer will also be asked the questions on the CMHS client-level data collection instrument. The one exception to this requirement is when a consumer had responded to these same questions within the past 30 days as part of a Reassessment.
Infrastructure development, prevention, and mental health promotion performance data are to be submitted quarterly by the grantee Project Directors through a web-based data entry system. Some programs may opt to keep track of their information using paper and pencil methods but are required to submit the data electronically within 30 days of the end of each quarter.
Each Services grantee collecting client-level data will have established its own procedures to collect baseline, periodic reassessment, and discharge data as part of the original protocol. For newly admitted consumers, baseline data collection would typically occur at the time of intake to the services program. All other data collection would occur as part of the normal course of service delivery, most likely by the primary provider assigned to the consumer. As noted, the timing of the periodic reassessment was chosen to coincide with normal clinical practice. Consumers are typically quite cooperative with grantee staff because of the relationship established during service provision. Since all participating grant programs will collect data at initial intake, considerable options also exist for non-respondent analysis and associated adjustments to the data such as weighting. Grantee Project Directors will be submitting infrastructure development, prevention, and mental health promotion data that documents grant activities; interviews are not a required component of the infrastructure, prevention, and mental health promotion data collection effort.
A relevant feature of the TRAC Reporting system is that it will automatically generate a report of when data submissions or interviews for existing consumer are due. Training on this and other features of the TRAC Reporting system will be provided to newly awarded grantees at national grantee meetings when possible. In addition to these training sessions, experts as well as selected grantees will be identified and asked to make presentations at national grantee meetings on the importance of quality and complete data collection, as well as TRAC system features to help facilitate consistency on consumer assessments at the appropriate intervals. Since these sessions are well attended by grantees, it is anticipated that these strategies will help to improve completion rates. The contractor also offers three annual refresher trainings via webinar to existing grantees to ensure the quality of the data collection and to help with grantee turnover.
All the data elements in the client-level data collection surveys were taken from established data collection instruments that have a history of use in the mental health field and have already been tested for validity and reliability, (i.e., the MHSIP, YSS-F, YSS, K-6, and ASSIST questions). In addition, for the domains that are not specific to mental health, CMHS has taken questions currently used by CSAT (OMB No. 0930-0208) that were drawn from widely used instruments and have been used for several years. These include three client-level domains (Employment/Education, Crime and Criminal Justice, and Stability in Housing) and one system-level domain (Access/Capacity), which depends on common demographics collected on consumers. The content of these questions was appropriate for use, but additional value options were defined to reflect issues specific to the populations served by CMHS. The benefits of using these measures include a history of use in monitoring the performance of CSAT grantees, the ability to conduct cross-Center comparisons, and use of measures previously approved by OMB.
The infrastructure, prevention, and mental health promotion data elements are drawn from these grant’s existing performance indicators and modified to allow consistent reporting for CMHS. A pilot of nine grant Project Directors was conducted using the attached instrument; results indicated these data are already part of routine data collection for most of the pilot participants or are consistent with their funded activities.
CMHS has contracted with Westat to provide support for the development and ongoing operational support for these data collection efforts, including statistical and analytic issues and the development of a web-based reporting the system. The Westat Project Director for this effort is: Bill Luckey, Ph.D., Vice President & Associate Director, Substance Abuse Research Group, phone: 301-610-4861. Jessica Taylor, Ph.D., (phone: 240-314-5852) will serve as the Deputy Project Director for Westat.
Crystal Blyler, Ph.D., (phone: 240-276-1910) will serve as the SAMHSA Project Officer responsible for receiving and approving contract deliverables. Sylvia Fisher, Ph.D., (phone: 240-276-1826) will serve as the Alternate Project Officer.
Attachment 1—Adult Client-level Measures
Attachment 2—Child Client-level Measures
Attachment 3—Infrastructure Development, Prevention, and Mental Health Promotion Performance Indicators
Attachment 4—Screen Shots
Attachment 5—Public Comments in Response to the Federal Register Notice
Attachment 6—IRB Exemption
1 Based on minimum wage.
2 Based on an estimate that it will be possible to conduct discharge interviews on 40 percent of those who leave the program.
3 Chart abstraction will be conducted on 100 percent of those discharged.
4 This is the maximum additional burden if all consumers complete the baseline and periodic reassessment interviews.
5 To be completed by grantee Project Directors, hence the higher hourly wage.
| File Type | application/msword |
| File Title | Transformation Accountability (TRAC) Reporting System |
| Author | Jessica Taylor |
| Last Modified By | SKING |
| File Modified | 2010-03-24 |
| File Created | 2010-03-24 |