SUPPORTING STATEMENT FOR THE
GOVERNMENT PERFORMANCE AND RESULTS ACT
CLIENT/PARTICIPANT OUTCOME MEASURES
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
B1. Respondent Universe and Sampling Methods
All SAMHSA grantees are required to collect and report certain data so that the Agency can meet its obligations under the Government Performance and Results Modernization Act of 2010 (GPRA).
In SAMHSA programs, which are often client level/participant interventions that are dramatically different from one group to another may have insufficient population receiving a specific intervention to justify a sample. Populations from each group may not be similar and would not be appropriate to infer general findings about the successes or failures of a program because of the uniqueness of each group. Within populations, sample sizes may be too small to properly sample, leading to large sample variance and errors in findings about the programs. In these cases, where programs differ from group to group, it is important to gather data sufficient to draw statistically accurate conclusions about how the programs are performing and about what characteristics of the program may matter to the success of the program.
In addition to the GPRA, data collected by grantees will be used to demonstrate how SAMHSA’s grant programs are reducing disparities in access, service use, and outcomes nationwide. To accomplish this, SAMHSA expects grantees to utilize their data to (1) identifying subpopulations (i.e., racial, ethnic, sexual/gender minority groups) vulnerable to health disparities and (2) implement strategies to decrease the differences in access, service use, and outcomes among those subpopulations. There will be subpopulations where sample size is too small to properly sample and where a census would be necessary obtain relevant and reliable outcome data.
Information collection procedures will vary by type of program. The client outcome measures for most providers will be extracted from previously established databases. Intake/baseline information is obtained by intake workers and/or counselors. For clients still in treatment 6 months later, the information will be obtained in the same way. In instances where clients are no longer in direct contact with the service provider, staff from the program will locate the clients and conduct the follow-up interviews. These interviews are to be conducted face-to-face.
Some programs collect their client information using paper and pencil methods. This project will not interfere with ongoing program operations. Programs will submit their data electronically via a web-based data entry process or upload process. The data for those clients with baseline, discharge, and follow-up data are matched using a unique encrypted client identifier.
Information data collection procedures will be the responsibility of individual grantees and may vary by type of program.
Some grantees have service providers conduct client-level baseline and follow-up assessments, while others have grant evaluators perform this function.
Some grantees may wish to collect client-level information using paper and pencil methods. SAMHSA will provide downloadable paper versions of the data collection instruments to facilitate this process. These grantees will then submit their data electronically via a web-based data entry process. The data for clients with both baseline and periodic reassessment data are matched using a unique encrypted client identifier developed by the grantee. Grantees will be clearly instructed not to use identifying information (i.e., social security number or initials) as the consumer identifier.
Required data collection points are:
BASELINE: For clients who have not previously been served by program, or who are returning to services following a discharge from the program, baseline data will be collected. For clients already enrolled in the program and continuing to receive services, administrative data should be submitted by the grantee within 30 days of initiating data collection. The timing of any subsequent data collection point(s) will be anchored to the baseline point the grantee indicates in this administrative record.
6 MONTHS FOLLOW-UP: SAMHSA requires client-level data collection every six months while the consumer is receiving SAMHSA-funded services. Ongoing periodic status review is viewed as consistent with good clinical practice.
DISCHARGE: Grantees must provide information on the type of discharge on all clients who are discharged. When the discharge is a planned event, the client will also be asked the questions on the client-level data collection instrument. The one exception to this requirement is when a client had responded to these same questions within the past 30 days as part of a Reassessment.
Each grantee also has its own plan for data collection, processing, data cleaning, control, and retention. Each plan describes how uniform data collection will be ensured, the time frame for conducting the assessments over the course of the project, and how participant protection will be assured. As mentioned above, these plans undergo peer review to ensure the adequacy and appropriateness of the study design and methods. The precise manner in which data will be collected and used depends on the specific grant program.
B3. Methods to Maximize Response Rates
Each grantee will have established its own client follow-up procedures as part of the original protocol. At the time of intake, information is typically obtained from clients to assist with locating them later. This includes information on current residents plus information on one or two other individuals who are likely to know where they are if they have re-located. In addition, some providers are adept at using other community resources to assist with locating clients. Clients are typically quite cooperative with provider staff because of the relationship established during treatment. Since all participating grant programs propose a census at initial intake, considerable options also exist for non-respondent analysis and associated adjustments to the data such as weighting.
Follow-up has been a challenge to some grantees given the remote locations that they serve and the challenge of locating clients as far out as 6 months. For grantees that have not been aware of the strategies they can employ to begin the follow-up process at intake, how to maintain contact with clients, and the importance of good locator forms, several strategies have been implemented to assist the grantees with follow-up. First, follow-up training is offered which assists grantees in learning about and conducting follow-up at their sites. This program is offered to all grantees and after the grantees are trained through the grantee orientation process, monthly follow-up trainings are offered for those that need additional training or for new project staff. Individual grantee technical assistance is also available for sites that need additional follow-up instruction. These group and individual trainings are conducted by follow-up experts. Each grantee receives a follow-up tracking manual at these trainings that may be used as a future reference. A second strategy provides the grantees with data status reports on how close they are to meeting their follow-up goals. These reports are available from the web-based system to the grantees and Government Project Officers for the grants they are responsible. A third strategy is the automatic, system generated notice of when follow-up interviews are due for each client/participant. A fourth strategy provides technical assistance at national meetings. Experts, including grantees, have been identified and asked to make presentations at national grantee meetings on how to conduct follow-up. These sessions are well attended by grantees.
SAMHSA will maintain a contract to provide technical assistance for grantees and Government Project Officers (GPOs) on the collection, management, and utilization of data. Specifically, the objectives include:
Surveying grantees to determine current training requirements;
Maintaining current records of all SAMHSA data collection requirements;
Providing training to grantees related to the collection, management, analysis, and utilization of data; and
Providing technical assistance to grantees regarding data in manner that supports sustainable practices Researching and disseminating information related to data collection, management, and utilization.
This contract will support the Agency’s ability to ensure conformance to program specifications.
Issues related to response rates, as well as other data collection issues, are discussed at grantee meetings in order for GPOs to identify problems and provide technical assistance. In addition, GPOs monitor data collection efforts and provide technical assistance to individual grantees as necessary. Because collection of outcome measures is a stipulation of the grants, it is anticipated that all grantees will comply (as appropriate). The participants at each site to whom these measures will be administered are all voluntary respondents; therefore, grantees cannot guarantee full cooperation on the part of participants. As part of the terms and conditions of the grant award, sites are required to use the outcome data system. Historically, however, participant response rates across grantee sites have averaged 80%.
B4. Test of Procedures
Most of the data elements in the data sets have been taken from established data collection instruments that have a long history of use in the substance abuse field and have already been tested for validity and reliability, (i.e., ASI). Revisions have been made to add the World Health Organization Quality of Life (WHO QOL-8) recovery measure questions to the GPRA instrument approved under a separate approval (OMB No. 0930–0208) which expires on March 31, 2019.
For the recovery measures, SAMHSA developed and piloted with a small number of individuals (OMB No. 0930-0342) a short 21-item instrument that was designed to capture all four of SAMHSA’s proposed dimensions of recovery – health, home, purpose, and community. That pilot allowed SAMHSA to establish usability of the recovery tool, and provided some evidence to support the use of the WHOQOL8 questions as a measure of recovery, but due to the small sample size, further pilot testing is needed in order to further analyze changes in response patterns over time. This project will provide the needed sample size by incorporating these questions into all CSAT discretionary grant programs which provide direct substance use treatment services to individuals. CSAT is adding the WHOQOL8 recovery measure questions because the other thirteen questions are addressed in the current approved tool and adding them would be duplicative. Data will be collected at two time points – at client intake and at six months post-intake. These are two points in time during which SAMHSA grantees routinely collect data on the individuals participating in their programs.
Feedback from the grantees also indicates that they routinely collect the same information requested of this data collection tool and some have integrated this tool into other tools that they routinely use to gather information. Some grantees report that they collect information in greater detail, (i.e., more response alternatives), but these are collapsed into standard categories.
B5. Statistical Consultants
Sarah Ndiangui
Government Project Officer
Center for Substance Abuse Treatment
SAMHSA
5600 Fishers Lane
Rockville, MD 20850
(240) 276-2918.
Robert Atanda, Ph.D.
Branch Chief, Performance Measurement Branch
Center for Substance Abuse Treatment
SAMHSA
5600 Fishers Lane
Rockville, MD 20850
(240) 276-2826
ATTACHMENTS
Attachment 1: CSAT GPRA Client Outcome Measures for Discretionary Programs and Instructions
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Department of Health and Human Services |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |