Final Supporting Statement B 9.19.18

SUPPORTING STATEMENT FOR

MENTAL HEALTH CLIENT/PARTICIPANT OUTCOME MEASURES

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B1. Respondent Universe and Sampling Methods

All Substance Abuse and Mental Health Services Administration (SAMHSA) grantees are required to collect and report certain data so that the Agency can meet its obligations under the Government Performance and Results Modernization Act of 2010 (GPRA). 

In general, SAMHSA programs are typically comprised of client level/participant interventions that are dramatically different from one another, often resulting in a wide variety of population groups served. Within populations served, sample sizes may be too small to properly sample, leading to large sample variance and errors in findings about the programs. In these cases, where programs differ from group to group, it is important to gather census-type data sufficient to draw statistically accurate conclusions about programs’ performance and about what characteristics of the program affect the success of the program.

While census-type data collection is usually necessary to obtain relevant and reliable data from SAMHSA grantees, there are situations in which sampling is a more appropriate technique. SAMHSA is proposing to use sampling techniques to collect client-level data from its Certified Community Behavioral Health Clinics Expansion (CCBHC-E) grant program. The purpose of this grant program is to increase access to, and improve the quality of, community behavioral health services through the expansion of CCBHCs. CCBHCs provide a comprehensive collection of services that create access, stabilize people in crisis, and provide needed treatment and recovery support services for those with the most serious and complex mental and substance use disorders. CCBHCs integrate additional services to ensure an approach to health care that emphasizes recovery, wellness, trauma-informed care, and physical-behavioral health integration. CCBHCs provide services to any individual, regardless of their ability to pay or their place of residence. Based on information provided in grant applications, SAMHSA anticipates CCBHC-E grantees will serve approximately 100,000 clients over two years. Because of the CCBHC-E program design, and the large numbers of individuals who will be served under this grant program, SAMHSA plans to utilize sampling techniques to collect client-level data from a subset of individuals receiving services under this grant program. SAMHSA will provide guidance to the CCBHC-E grantees to utilize simple random sampling techniques to collect client-level data from 10 percent of each clinic’s clients, rather than requiring clinics to collect data from all clients. SAMHSA believes this strategy will reduce grantee burden and produce cost savings to the government, while yielding enough data to meet GPRA requirements.

B2. Information Data Collection Procedures

Information data collection procedures will be the responsibility of individual grantees and may vary by type of program.

Client-level data collection

Some grantees have service providers conduct client-level baseline and follow-up assessments, while others have grant evaluators perform this function.

Some grantees may wish to collect their client-level information using paper and pencil methods. SAMHSA will provide downloadable paper versions of the data collection instruments to facilitate this process. Grantees will then submit their data electronically via a web-based data entry process or upload process. The data for those clients with both baseline and periodic reassessment data are matched using a unique encrypted client identifier developed by the grantee. Grantees will be clearly instructed not to use identifying information (i.e., social security number) as the patient identifier.

Required data collection points are:

BASELINE: For clients who have not previously been seen by the grantee, baseline data will be collected at admission. For clients already enrolled in the program and continuing to receive services, administrative data should be submitted by the grantee within 30 days of initiating data collection. The timing of any subsequent data collection point(s) will be anchored to the baseline point the grantee indicates in this administrative record.

PERIODIC ASSESSMENT: SAMHSA requires client-level data collection every six months while the client is receiving SAMHSA-funded services. Ongoing periodic status review is viewed as consistent with good clinical practice. SAMHSA is currently seeking approval for quarterly assessment of physical health indicators collected from adult and child/adolescent clients.

DISCHARGE: Grantees must provide information on the type of discharge on all clients who are discharged. When the discharge is a planned event, the client will also be asked the questions on the CMHS client-level data collection tool. The one exception to this requirement is when a client had responded to these same questions within the past 30 days as part of a reassessment.

Infrastructure development, prevention, and mental health promotion performance data collection

Infrastructure development, prevention, and mental health promotion performance data are to be submitted quarterly by the grantee Project Directors through a web-based data entry system. Some programs may opt to keep track of their information using paper and pencil methods but are required to submit the data electronically within 30 days of the end of each quarter.

B3. Methods to Maximize Response Rates

Each grantee will have established its own procedures to collect baseline, periodic reassessment, and discharge data as part of the original protocol. For newly admitted clients, baseline data collection would typically occur at the time of intake. All other data collection would occur as part of the normal course of treatment, most likely by the primary counselor or clinician assigned to the client. As noted, the timing of the periodic reassessment was chosen to coincide with normal clinical practice. Clients are typically quite cooperative with grantee staff because of the relationship established during treatment. Since all participating grant programs will collect data at initial intake, considerable options also exist for non-respondent analysis and associated adjustments to the data such as weighting. Grantee Project Directors will be submitting infrastructure development, prevention, and mental health promotion data that documents grant activities; interviews are not a required component of the infrastructure, prevention, and mental health promotion data collection effort

A relevant feature of the SAMHSA Performance Accountability and Reporting System is that it will automatically generate notices of when periodic assessment interviews are due for each client. Training on this and other features of the web-based system will be provided to newly awarded grantees at national grantee meetings, when possible. In addition to these training sessions, experts and selected grantees will be identified and asked to present at national grantee meetings on quality data collection, and web-based system features to help facilitate consistent client assessment. Since these sessions are well attended by grantees, it is anticipated that these strategies will help to improve completion rates.

B4. Tests of Procedures

Many of the data elements in the proposed data sets have been taken from established data collection instruments that have a history of use in the mental health field and have already been tested for validity and reliability, (i.e., Mental Health Statistics Improvement Program Survey, Youth Services Survey for Families, and Youth Services Survey for Youth). In addition, for the domains that are not specific to mental health, CMHS has taken questions currently used by SAMHSA’s Center for Substance Abuse Treatment (CSAT) (Office of Management and Budget No. 0930-0208) that were drawn from widely used instruments and have been used for several years. These include three client-level domains (Employment/Education, Crime and Criminal Justice, and Stability in Housing) and one system-level domain (Access/Capacity), which depends on common demographics collected on clients. The content of these questions was appropriate for use, but additional value options were defined to reflect issues specific to the populations served by CMHS. The benefits of using these measures include a history of use in monitoring the performance of CSAT grantees, the ability to conduct cross-Center comparisons, and use of measures previously approved by OMB.

B5. Statistical Consultant

Mark Jacobsen, Ph.D.
Public Health Analyst

DHHS/SAMHSA
Center for Mental Health Services
Office of Program Analysis and Coordination

5600 Fishers Lane, 14E57B

Rockville, MD 20857

240-276-1826

3