0285-ss_b-cmhs 9-7-16

Transformation Accountability (TRAC) Reporting System

Supporting Statement

Statistical Methods

1.Respondent Universe and Sampling Methods

All SAMHSA grantees are required to collect and report certain data so that the Agency can meet its obligations under their funding agreements. 

SAMHSA programs, which are often client level/participant interventions that are dramatically different from one group to another, may have insufficient populations receiving a specific intervention to justify a sample. Populations from each group may not be similar and it would not be appropriate to infer general findings about the successes or failures of a program because of the uniqueness of each group. In addition, population sizes may be too small to sample from, leading to large sample variance and errors in findings about the programs. In these cases, where programs differ from group to group, it is important to gather data sufficient to draw statistically accurate conclusions about how the programs are performing and about what characteristics of the program may matter to the success of the program.

In addition to the GPRA, data collected by grantees will be used to demonstrate how SAMHSA’s grant programs are reducing disparities in access, service use, and outcomes nationwide. To accomplish this, SAMHSA expects grantees to utilize their data to (1) identify subpopulations (i.e., racial, ethnic, sexual/gender minority groups) vulnerable to health disparities and (2) implement strategies to decrease the differences in access, service use, and outcomes among those subpopulations. There will be subpopulations where sample size is too small to properly sample and where a census would be necessary obtain relevant and reliable outcome data.

All CMHS-funded discretionary grantees that provide direct services or engage in infrastructure development, prevention, or mental health promotion activities are required to participate in this data collection effort and will submit data based on their conducted activities. The table below indicates the grant programs for each of the TRAC data collection efforts:

Table 1. Data Collection Effort by CMHS-funded Programs

CMHS-funded Program	Total Number of Grants	Client-level	Infrastructure Development, Prevention, and Mental Health Promotion
Assisted Outpatient Treatment	15	Yes	Yes
Circles of Care	11	No	Yes
Comprehensive Community Mental Health Services for Children and their Families Program	79	Yes	Yes
Garrett Lee Smith Suicide Prevention Resource Center	1	No	Yes
Garrett Lee Smith Campus Suicide Prevention Grant Program	55	No	Yes
Garrett Lee Smith State/Tribal Suicide Prevention Grant Program	42	No	Yes
Healthy Transitions Initiative	1	Yes	Yes
Historically Black Colleges & Universities National Resource Center	1, with 20 sub-grantees	No	Yes
Linking Actions for Unmet Needs in Children’s Mental Health	36	No	Yes
National Strategy for Suicide Prevention	4	No	Yes
National Suicide Prevention Lifeline	1	No	Yes
NCTSI Treatment & Service Centers	25	Yes	Yes
NCTSI Community Treatment Centers	56	Yes	Yes
NCTSI National Coordinating Center	1	No	Yes
Mental Health First Aid	69
Mental Health Transformation State Incentive Grant	7	Yes	Yes
Minority AIDS/HIV Services Collaborative Program	34	Yes	No
Minority Fellowship Program	11	No	Yes
Older Adults Targeted Capacity Expansion	10	Yes	No
Primary and Behavioral Health Care Integration	80	Yes	Yes
Project AWARE State Grants	27	No	Yes
Safe Schools/Health Students Initiative	7	No	Yes
Statewide Consumer Network Grants	17	No	Yes
Statewide Family Networks Grants	33	No	Yes
Suicide Lifeline Crisis Center FUP Grants	6	No	Yes
Tribal Behavioral Health Grants	90	No	Yes
Total	719	307	412

2.Information Collection Procedures

Information data collection procedures will be the responsibility of individual grantees and may vary by type of program.

Client-level data collection

Some grantees have service providers conduct client-level baseline and follow-up assessments, while others have grant evaluators perform this function.

Some grantees may wish to collect client-level information using paper and pencil methods. CMHS will provide downloadable paper versions of the data collection instruments to facilitate this process. These grantees will then submit their data electronically via a web-based data entry process. The data for those consumers with both baseline and periodic reassessment data are matched using a unique encrypted consumer identifier developed by the grantee. Grantees will be clearly instructed not to use identifying information (i.e., social security number or initials) as the consumer identifier.

Required data collection points are:

BASELINE: For consumers who have not previously been seen by the grantee, baseline data will be collected at admission. For consumers already enrolled in the program and continuing to receive services, administrative data should be submitted by the grantee within 30 days of initiating TRAC data collection. The timing of any subsequent data collection point(s) will be anchored to the baseline point the grantee indicates in this administrative record.

REASSESSMENT: CMHS requires client-level data collection every six months while the consumer is receiving CMHS-funded services. Ongoing periodic status review is viewed as consistent with good clinical practice.

DISCHARGE: Grantees must provide information on the type of discharge on all consumers who are discharged. When the discharge is a planned event, the consumer will also be asked the questions on the CMHS client-level data collection instrument. The one exception to this requirement is when a consumer had responded to these same questions within the past 30 days as part of a Reassessment.

Infrastructure development, prevention, and mental health promotion performance data collection

Infrastructure development, prevention, and mental health promotion performance data are to be submitted quarterly by the grantee Project Directors through a web-based data entry system. Some programs may opt to keep track of their information using paper and pencil methods but are required to submit the data electronically within 30 days of the end of each quarter.

3.Methods to Maximize Response Rates

Each Services grantee collecting client-level data will have established its own procedures to collect baseline, periodic reassessment, and discharge data as part of the original protocol. For newly admitted consumers, baseline data collection would typically occur at the time of intake to the services program. All other data collection would occur as part of the normal course of service delivery, most likely by the primary provider assigned to the consumer. As noted, the timing of the periodic reassessment was chosen to coincide with normal clinical practice. Consumers are typically quite cooperative with grantee staff because of the relationship established during service provision. Since all participating grant programs will collect data at initial intake, considerable options also exist for non-respondent analysis and associated adjustments to the data such as weighting. Grantee Project Directors will be submitting infrastructure development, prevention, and mental health promotion data that documents grant activities; interviews are not a required component of the infrastructure, prevention, and mental health promotion data collection effort.

A relevant feature of the TRAC Reporting system is that it will automatically generate a report of when data submissions or interviews for existing consumer are due. Training on this and other features of the TRAC Reporting system will be provided to newly awarded grantees at national grantee meetings when possible. In addition to these training sessions, experts as well as selected grantees will be identified and asked to make presentations at national grantee meetings on the importance of quality and complete data collection, as well as TRAC system features to help facilitate consistency on consumer assessments at the appropriate intervals. Since these sessions are well attended by grantees, it is anticipated that these strategies will help to improve completion rates. The contractor also offers three annual refresher trainings via webinar to existing grantees to ensure the quality of the data collection and to help with grantee turnover.

4.Tests of Procedures

All the data elements in the client-level data collection surveys were taken from established data collection instruments that have a history of use in the mental health field and have already been tested for validity and reliability, (i.e., the Mental Health Statistics Improvement Program, Youth Services Survey for Families, Youth Services Survey, Kessler-6, World Health Organization Quality of Life 8, and Alcohol Smoking and Substance Involvement Screening Test questions). In addition, for the domains that are not specific to mental health, CMHS has taken questions currently used by SAMHSA’s Center for Substance Abuse Treatment (CSAT) (OMB No. 0930-0208) that were drawn from widely used instruments and have been used for several years. These include three client-level domains (Employment/Education, Crime and Criminal Justice, and Stability in Housing) and one system-level domain (Access/Capacity), which depends on common demographics collected on consumers. The content of these questions was appropriate for use, but additional value options were defined to reflect issues specific to the populations served by CMHS. The benefits of using these measures include a history of use in monitoring the performance of CSAT grantees, the ability to conduct cross-Center comparisons, and use of measures previously approved by OMB.

The infrastructure, prevention, and mental health promotion data elements are drawn from these grant’s existing performance indicators and modified to allow consistent reporting for CMHS. A pilot of nine grant Project Directors was conducted using the attached instrument; results indicated these data are already part of routine data collection for most of the pilot participants or are consistent with their funded activities.

For the recovery measures, SAMHSA developed and piloted with a small number of individuals (OMB No. 0930-0342) a short 21-item instrument that was designed to capture all four of SAMHSA’s proposed dimensions of recovery – health, home, purpose, and community. That pilot allowed SAMHSA to establish usability of the recovery tool, but due to the small sample size, further pilot testing is needed in order to analyze the psychometric properties and change in response over time to determine if the measurements are stable over time. This project will provide the needed sample size by using it with all CMHS discretionary grant programs which provide direct mental health services to individuals. The recovery measure is comprised of questions from the WHO QOL-8 and SAMHSA’s existing set of GPRA measures. Data will be collected at two time points – at client intake and at six months post-intake. These are two points in time during which SAMHSA grantees routinely collect data on the individuals participating in their programs.

5.Statistical Consultants

CMHS is in the process of awarding a contract to provide support for the development and ongoing operational support for these data collection efforts, including statistical and analytic issues and the development of a web-based reporting the system. Mark P. Jacobsen, Ph.D. (phone: 240-276-1826) will serve as the SAMHSA Project Officer responsible for receiving and approving contract deliverables. Kirstin Painter, Ph.D., (phone: 240-276-1932) will serve as the Alternate Project Officer.

List of Attachments

Attachment 1—Adult Client-level Measures

7