SUPPORTING STATEMENT FOR THE
GOVERNMENT PERFORMANCE AND RESULTS ACT
CLIENT/PARTICIPANT OUTCOME MEASURE
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
B1. Respondent Universe and Sampling Methods
All Substance Abuse and Mental Health Services (SAMHSA) grantees are required to collect and report certain data so that the Agency can meet its obligations under the Government Performance and Results Modernization Act of 2010 (GPRA).
SAMHSA programs, which are often comprised of client level/participant interventions that are dramatically different from one another, may have insufficient population receiving a specific intervention to justify a sample. Populations receiving program interventions may not be similar and, thus it would not be appropriate to infer general findings about the successes or failures of a program because of the uniqueness of populations served. Within populations served, sample sizes may be too small to properly sample, leading to large sample variance and errors in findings about the programs. In these cases, where programs differ from group to group, it is important to gather data sufficient to draw statistically accurate conclusions about programs’ performance and about what characteristics of the program affect the success of the program.
In addition to the GPRA, data collected by grantees will be used to demonstrate how SAMHSA’s grant programs are reducing disparities in access, service use, and outcomes nationwide. To accomplish this, SAMHSA expects grantees to utilize their data to (1) identifying subpopulations (i.e., racial, ethnic, sexual/gender minority groups) vulnerable to health disparities; and, (2) implement strategies to decrease the differences in access, service use, and outcomes among those subpopulations. There will be subpopulations where sample size is too small to properly sample and where a census would be necessary obtain relevant and reliable data.
Information collection procedures will vary by type of program. For most providers, some information will be extracted from previously established databases. Intake/baseline information is obtained by intake workers and/or counselors. For clients still in treatment six months after intake, the information will be obtained in the same way. In instances where clients are no longer in direct contact with the service provider, staff from the program will locate the clients and conduct the follow-up interviews.
Some programs collect their client information using paper-and-pencil methods. This project will not interfere with ongoing program operations. Programs will submit their data electronically via a web-based data entry system, the SAMHSA’s Performance Accountability and Reporting System (SPARS), using a unique, encrypted client identifier for each client. This unique client identifier will be used to match client data across each data collection time point (baseline, follow-up, and discharge). Information data collection procedures will be the responsibility of individual grantees and may vary by type of program.
Some grantees have service providers conduct client-level baseline and follow-up assessments, while others have grant evaluators perform this function.
Some grantees may wish to collect client-level information using paper and pencil methods. SAMHSA will provide downloadable paper versions of the data collection instrument to facilitate this process. These grantees will then submit their data electronically via a web-based data entry process. The data for clients with both baseline and periodic reassessment data are matched using the SPARS ID. Grantees will be clearly instructed not to use identifying information (i.e., social security number or initials) as the client identifier.
Required data collection points are:
BASELINE: For clients who have not previously been served by program, or who are returning to services following a discharge from the program, baseline data will be collected. For clients already enrolled in the program and continuing to receive services, data should be submitted by the grantee within 30 days of initiating data collection. The timing of any subsequent data collection point(s) will be anchored to the baseline point the grantee indicates in the administrative record.
SIX MONTH FOLLOW-UP: SAMHSA requires client-level data collection six months after baseline/intake. Ongoing periodic status review is viewed as consistent with good clinical practice.
DISCHARGE: Grantees must provide information on the type of discharge on all clients who are discharged. When the discharge is a planned event, the client will also be asked the questions on the client-level data collection instrument. The one exception to this requirement is when a client had responded to these same questions within the past 30 days as part of the six-month follow-up.
Each grantee may have its own plan for data collection, processing, data cleaning, control, and retention. Each plan should describe how uniform data collection will be ensured, the time frame for conducting the data collection over the course of the project, and how participant protection will be assured. These plans undergo peer review to ensure the adequacy and appropriateness of the study design and methods. The precise manner in which data will be collected and used depends on the specific grant program.
B3. Methods to Maximize Response Rates
Each grantee will have established its own client follow-up procedures as part of the original protocol. At the time of intake, information is typically obtained from clients to assist with locating them later. This includes information on current residence and contact information for one or two other individuals who are likely to know where they are if they have re-located. In addition, some providers are adept at using other community resources to assist with locating clients. Clients are typically quite cooperative with provider staff because of the relationship established during treatment. Since all participating grant programs propose a census at initial intake, considerable options also exist for non-respondent analysis and associated adjustments to the data such as weighting.
Follow-up has been a challenge for some grantees, given the remote locations that they serve and the challenge of locating clients six months after intake. For grantees that have not been aware of the strategies they can employ to begin the follow-up process at intake, how to maintain contact with clients, and the importance of good locator forms, several strategies have been implemented to assist the grantees with follow-up. First, follow-up training is offered to assist grantees in learning about and conducting follow-up at their sites. This program is offered to all grantees and after the grantees are trained through the grantee orientation process, monthly follow-up trainings are offered for those that need additional training or for new project staff. Individual grantee technical assistance is also available for sites that need additional follow-up instruction. These group and individual trainings are conducted by follow-up experts. Each grantee receives a follow-up tracking manual at these trainings that may be used as a future reference. A second strategy provides the grantees with data status reports on how close they are to meeting their follow-up goals. These reports are available from the web-based system to the grantees and Government Project Officers (GPOs) for the grants they are responsible. A third strategy is the automatic, system-generated notice of when follow-up interviews are due for each client/participant. A fourth strategy provides technical assistance at national meetings. Experts, including grantees, have been identified and asked to make presentations at national grantee meetings on how to conduct follow-up. These sessions are well attended by grantees.
SAMHSA will maintain a contract to provide technical assistance for grantees and GPOs on the collection, management, and utilization of data. Specifically, the objectives include:
Surveying grantees to determine current training requirements;
Maintaining current records of all SAMHSA data collection requirements;
Providing training to grantees related to the collection, management, analysis, and utilization of data; and
Providing technical assistance to grantees regarding data in manner that supports sustainable practices Researching and disseminating information related to data collection, management, and utilization.
This contract will support the Agency’s ability to ensure conformance to program specifications.
Issues related to response rates, as well as other data collection issues, are discussed at grantee meetings in order for GPOs to identify problems and provide technical assistance. In addition, GPOs monitor data collection efforts and provide technical assistance to individual grantees as necessary. Because collection of outcome measures is a stipulation of the grants, it is anticipated that all grantees will comply (as appropriate). The participants to whom these measures will be administered are all voluntary respondents; therefore, grantees cannot guarantee full cooperation on the part of participants. As part of the terms and conditions of the grant award, sites are required to use the web-based data entry system.
B4. Test of Procedures
Most of the items on the existing data collection instrument used in this information collection have come from a tool that has been used extensively in the substance abuse field and has already been tested for validity and reliability (the Addiction Severity Index). The data collection instrument has been has been approved under a separate approval (OMB No. 0930–0208), which expires on January 31, 2020. Revisions have been made to add 13 questions to the existing instrument. Individual grantees will be responsible for responding to no more than four new questions (up to three program-specific questions in addition to the behavioral health diagnosis question).
The 13 questions added to the data collection instrument were developed by SAMHSA to understand program-specific services or activities delivered, and associated outcomes. Data will be collected at three time points – at intake, six months post-intake, and discharge. These are points in time during which SAMHSA grantees routinely collect data on the individuals participating in their programs.
B5. Statistical Consultants
Christopher Jones, PharmD. M.P.H
Director, National Mental Health and Substance Use Policy Lab
5600 Fishers Lane
Rockville, MD
ATTACHMENTS
Attachment 1: CSAT GPRA Client Outcome Measure for Discretionary Programs and Instructions
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Department of Health and Human Services |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |