0208-06-ss-11-28-06

SUPPORTING STATEMENT FOR THE

GOVERNMENT PERFORMANCE AND RESULTS ACT

CLIENT/PARTICIPANT OUTCOME MEASURES

A. JUSTIFICATION

1. Circumstances of Information Collection

The Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Substance Abuse Treatment (CSAT) is requesting from the Office of Management and Budget (OMB) approval for the revision of data collection activities for the Government Performance and Results Act Client/Participant Outcome Measures (OMB No. 0930–0208), that expires on January 31, 2007. There are no changes to the instrument used for data collection for CSAT. In regard to the Center for Mental Health Services (CMHS), the burden will be transferred from this data collection to its own separate OMB clearance. The 30-day Federal Register Notice for the National Outcome Measures (NOMs) for Consumers Receiving Mental Health Services was published on Friday, December 15, 2006 (71 FR 75567) and the package was submitted to OMB on January 4, 2007. In regard to the Center for Substance Abuse Prevention (CSAP), the burden will be reduced annually as those grantees funded prior to FY 2006 phase out. Grantees funded in FY 2006 and later are expected to use the NOMs tool approved by OMB in December 2005 (OMB No. 0930-0230).

Approval of this information collection will allow SAMHSA to continue to meet Government Performance and Results Act of 1993 (GPRA) reporting requirements that quantify the effects and accomplishments of its discretionary grant programs which are consistent with OMB guidance. (See Attachment 1 for the previously approved CSAT tool and instructions and Attachment 2 for the previously approved CSAP tool and instructions.)

In order to carry out section 1105(a) (29) of the GPRA, SAMHSA is required to prepare a performance plan for its major programs of activity. This plan must:

a) Establish performance goals to define the level of performance to be achieved by a program activity;

b) Express such goals in an objective, quantifiable, and measurable form;

c) Briefly describe the operational processes, skills and technology, and the human, capital, information, or other resources required to meet the performance goals;

d) Establish performance indicators to be used in measuring or assessing the relevant outputs, service levels, and outcomes of each program activity;

e) Provide a basis for comparing actual program results with the established performance goals; and

f) Describe the means to be used to verify and validate measured values.

SAMHSA’s legislative mandate is to increase access to high quality prevention and treatment services and to improve outcomes. Its mission is to improve the quality and availability of treatment and prevention services for substance abuse and mental illness. To support this mission, the Agency’s overarching goals are:

1) Accountability—Establish systems to ensure program performance measurement and accountability

2) Capacity—Build, maintain, and enhance mental health and substance abuse infrastructure and capacity

3) Effectiveness—Enable all communities and providers to deliver effective services

Each of these key goals complements SAMHSA’s legislative mandate. All of SAMHSA’s programs and activities are geared toward the achievement of these goals and GPRA performance monitoring is a collaborative and cooperative aspect of this process.

SAMHSA is striving to coordinate the development of these goals with other ongoing performance measurement development activities, for example, development of performance measures for reporting of activities under the block grant programs. This information collection is needed to provide objective data to demonstrate SAMHSA’s monitoring and achievement of its mission and goals.

Another coordination effort is with the Office of National Drug Control Policy (ONDCP). In addition to examining SAMHSA’s GPRA performance targets, the data will be useful in directly addressing the demand reduction goals and objectives outlined in ONDCP’s National Drug Control Strategy. SAMHSA has participated in the development of the ONDCP strategy and has provided direct programmatic support to their two-year goals and will continue to provide support to the five-year goals. The following two-year goals have been met and exceeded:

• Two-Year Goals Met:

— Goal 1: A 10 percent reduction in current use of illegal drugs by the 12 through 17 year age group.

— Goal 2: A 10 percent reduction in current use of illegal drugs by adults age 18 years and older.

SAMHSA programs will continue to contribute to the following five-year goals:

• Five-Year Goals:

— Goal 1: A 25 percent reduction in current use of illegal drugs by the 12 through 17 year age group.

— Goal 2: A 25 percent reduction in current use of illegal drugs by adults age 18 years and older.

2. Purpose and Use of Information

Outcome data reflect the Agency’s desire for consistency in data collected within the Agency. SAMHSA has implemented specific performance domains called National Outcome Measures (NOMs) to assess the accountability and performance of its discretionary and formula grant programs. These domains represent SAMHSA CSAT’s focus on the factors that contribute to the success of substance abuse treatment. CSAP’s NOMs have been approved under a separate submission. The CSAT domains include clients/participants who are adults and children/adolescents under age 18 years. The CSAT Client/Participant Outcome Measures will address the following performance domains:

• Abstinence from Drug / Alcohol Use

• Employment / Education

• Crime and Criminal Justice

• Family and Living Conditions

• Social Connectedness

• Access / Capacity

• Retention

• Cost effectiveness information is taken from the grant application.

SAMHSA uses the performance measures to report on the performance of its discretionary services grant programs. The performance measures information is used by individuals at three different levels: the SAMHSA administrator and staff, the Center administrators and government project officers, and grantees:

SAMHSA Level—The information is used to inform the administration of the performance of the programs funded through the Agency. The performance is based on the goals of the grant program and includes the NOMs. This information serves as the basis of the annual GPRA report to Congress contained in the Justifications of Budget Estimates.

Center Level—In addition to exploring the performance of the various programs, the information is used to monitor and manage individual grant projects within each program. The information informs the government project officers of the projects staff’s abilities to meet their individual goals. The information has been used by government project officers to make funding continuation decisions.

Grantee Level—In addition to monitoring performance outcomes, the grantee staff uses the information to improve the quality of treatment and prevention services that are provided to clients within their projects.

SAMHSA and each of its Centers will use the data for annual reporting required by GPRA and for NOMs comparing baseline with discharge and follow-up data. GPRA requires that SAMHSA’s report for each fiscal year include actual results of performance monitoring for the three preceding fiscal years. The additional information collected through this process will allow SAMHSA to report on the results of these performance outcomes as well as be consistent with the specific performance domains that SAMHSA is implementing as the NOMs, to assess the accountability and performance of its discretionary and formula grant programs. The CSAT client-level data items were initially identified from widely used data collection instruments (i.e., the Addiction Severity Index [ASI]; the 2000 National Household Survey on Drug Abuse; the McKinney Homeless Program reporting system; and the Risk Assessment Behavior Battery.)

Attachment 3 is a matrix showing the interrelationships among the SAMHSA goals and data items, and identifies the source of the items. There are no changes requested.

Among the measures delineated in SAMHSA’s annual, fiscal year GPRA Plan are a core set of client/participant outcome measures to be applied, as appropriate, to all of SAMHSA’s discretionary grant programs providing client/participant services. SAMHSA has established these standardized client/participant outcome measures in order to capture this essential client/participant-level information. The data set collected under this approval comprises items typically collected by substance abuse prevention and substance abuse treatment providers at the client/participant level. These discretionary programs are distributed between SAMHSA’s three centers: CMHS, CSAP, and CSAT. The measures for the agencies that are the subject of this information collection, CSAP and CSAT, are presented here.

CSAP: Substance Abuse Prevention Measures (for grants funded prior to FY 2006)

1. Over the past month, the percentage of youth:

a) Using substances declined for those receiving services compared to the national average or project baselines.

b) Strongly disapproving of substance use increased for those receiving services compared to the national average or project baselines.

c) Perceiving personal/health risks associated with the consequences of substance abuse increased for those receiving services compared to the national average or project baselines.

d) Receiving services reported an increase in the age of first use compared to the national average or project baselines.

e) Receiving services reported a decline in intention to use substances compared to the national average or project baselines.

2) Over the past month, the percentage of adults:

a) Using illegal drugs declined for those receiving services compared to the national average or project baselines.

b) Strongly disapproving of substance use increased for those receiving services compared to the national average or project baselines.

c) Perceiving personal/health risks associated with the consequences of substance abuse/misuse increased for those receiving services compared to the national average or project baselines.

d) Receiving services reported an increase in the age of first use compared to the national average or project baselines.

e) Receiving services reported a decline in intention to use substances compared to the national average or project baselines.

CSAT: Substance Abuse Treatment Measures

1) Over the past year, the percentage of adults:

a) Who were currently employed or engaged in productive activities increased for those receiving services compared to the national average or project baselines.

b) Who had a permanent place to live in the community increased for those receiving services compared to the national average or project baselines.

c) Who had reduced involvement with the criminal justice system increased for those receiving services compared to the national average or project baselines.

d) Who had no past month use of illegal drugs or misuse of prescription drugs increased for those receiving services compared to the national average or project baselines.

e) Who increased retention in the program/services compared to the national average or project baselines.

f) Who increased social connectedness to family and friends compared to the national average or project baselines.

g) Who increased access to services compared to the national average or project baselines.

An additional measure is for those adults:

Who experienced reduced alcohol or illegal drug related health, behavior, or social consequences (including the misuse of prescription drugs), increased for those receiving services compared to the national average or project baselines.

2) Over the past year, the percentage of children/adolescents under age 18:

a) Who were attending school increased for those receiving services compared to the national average or project baselines.

b) Who were residing in a stable living environment increased for those receiving services compared to the national average or project baselines.

c) Who had no involvement in the juvenile justice system increased for those receiving services compared to the national average or project baselines.

d) Who had no past month use of alcohol or illegal drugs (population data limited to 12 through 17 year olds) increased for those receiving services compared to the national average or project baselines.

e) Who increased retention in the program/services compared to the national average or project baselines.

f) Who increased social connectedness to family and friends compared to the national average or project baselines.

g) Who increased access to services compared to the national average or project baselines.

An additional measure is for those children/adolescents under age 18:

The percentage of youth (population data limited to 12 through17 year olds) who experienced no substance abuse related health, behavior, or social consequences increased for those receiving services compared to the national average or project baselines.

The Center for Substance Abuse Prevention (CSAP) is not changing these measures for the grants funded prior to FY 2006 that are included in this submission. As previously mentioned, CSAP is phasing out these measures and is requiring for grantees initially funded in FY 2006 and beyond, the NOMs measures under OMB 0930-0230.

Based on current funding and planned fiscal year 2007 notice of funding announcements (NOFA), the CSAT programs that will use these measures in fiscal years 2007 through 2010 include: the Addictions Treatment for Homeless; Assertive Adolescent Family Treatment; Effective Adolescent Treatment; HIV/AIDS Outreach; Pregnant and Postpartum Women; Recovery Community Services Program – Services; Rehabilitation and Restitution Services; Strengthening Communities - Youth; Screening and Brief Intervention and Referral to Treatment (SBIRT) and Campus Screening and Brief Intervention (SBI); Targeted Capacity Expansion (TCE); TCE/HIV; Treatment Drug Court; and the Youth Offender Reentry Program.

3. Use of Information Technology

Most programs collect their client/participant information using a variety of methods from paper and pencil to electronic methods. This project will not interfere with ongoing program collection operations that facilitate information collection at each site.

A web-based data collection and entry system has been developed through CSAT and is available to all programs for data collection. This web-based system allows for easy data entry, submission, and reporting to all those who have access to the system. Levels of access have been defined for users based on their authority and responsibilities regarding the data and reports. Access to the data and reports is limited to those individuals with a username and password.

A few programs submit their data electronically through an upload process. This facilitates the submission of data while avoiding duplication of the data entry process. Programs that collect these data for other purposes are spared the additional collection burden.

Electronic submission of the data promotes enhanced data quality. With built-in data quality checks, easy access to data outputs and reports, users of the data can feel confident about the quality of the output. The electronic submission also promotes immediate access to the dataset. Once the data are put into the web-based system, it is available for access, review, and reporting by all those with access to the system from Center staff to the grantee staff. Copies of the CSAT screens that data entry staff see when entering data are in Attachment 4.

4. Effort to Identify Duplication

The items collected are necessary in order to assess grantee performance. They are collected for the purposes of this project and where available may be used for this data collection activity.

5. Involvement of Small Entities

Individual grantees vary from small entities through large provider organizations. Every effort has been made to minimize the number of data items collected from programs to the least number required to accomplish the objectives of the effort and to meet GPRA reporting requirements and therefore, there is no significant impact involving small entities.

6. Consequences If Information Collected Less Frequently

The data collection points remain unchanged from the previous submission. Substance abuse prevention programs collect participant information at pre- and post-intervention, and then at a third point in time, either at 6 or 12 months post-intervention as appropriate to the program. For substance abuse treatment programs, the data collection points of intake, discharge, and 6-months post intake are part of regular program activity.

These are generally accepted intervals for client/participant assessment and the participants will be asked to respond to the items according to this schedule. The adolescent substance abuse treatment grantees are required to collect information additionally at three months post-intake due to the migratory nature of adolescents. It is more difficult to locate adolescents than adults and, therefore, locating them more frequently and closer to their intake date should increase their follow-up rates. The data will be reported to SAMHSA on an annual basis in keeping with the GPRA requirements for annual reporting.

7. Consistency with the Guidelines in 5 CFR 1320.5(d) (2)

This information collection fully complies with 5 CFR 1320.5(d) (2).

8. Consultation Outside the Agency

The notice required by 5 CFR 1320.8(d) was published in the Federal Register on September 14, 2006 (71 FR 54298). No comments were received in response to this notice.

Since CSAP is not making changes, feedback for this center was not solicited at this time. Since CSAT is not making changes, feedback for this center was not solicited.

9. Payment to Respondents

Grantees are asked to budget for data collection in their grant applications and individual grantees are not prohibited from providing payments to their respondents for follow-up, which is customary practice in the field. If the grantees do provide payment for the follow-up, the maximum incentive is $20.00 or the equivalent in coupons, transportation tokens, or other items per follow-up.

Survey research literature suggests that monetary incentives have a strong positive effect on response rates and no known adverse effect on reliability. In particular, substance abuse research has shown improved response rates when remuneration is offered to respondents. Substance abusers are typically a harder-to-reach population for whom out-of-pocket costs of participation (e.g., transportation, child care) are significant barriers.

10. Assurance of Confidentiality

SAMHSA has statutory authority to collect data under the Government Performance and Results Act (Public Law 1103(a), Title 31) and is subject to the Privacy Act for the protection of data. Substance abuse treatment providers are subject to the federal regulations for the confidentiality of alcohol and substance abuse patient records (42 CFR Part 2) which govern the protection of patient identifying data. In some cases, these same providers meet the definition of a HIPAA covered entity and are additionally subject to the Privacy Rule (45 CFR Parts 160 and 164) for the protection of individually identifiable data.

Client/participant treatment records in federally assisted alcohol and drug abuse programs are protected under 42 CFR Part 2 (OMB No. 0930-0092). SAMHSA and its contractors will not receive identifiable client/participant records. Provider-level information will be aggregated to, at the least, the level of the grant/cooperative agreement-funding announcement.

Grantees and all other potential respondents will be assured that confidentiality is maintained throughout data collection (to the extent permitted by law). All data will be closely safeguarded, and no institutional or individual identifiers will be used in reports. Only aggregated data will be reported.

11. Questions of a Sensitive Nature

SAMHSA’s mission is to improve the quality and availability of prevention, early intervention, treatment, and rehabilitation services for substance abuse and mental illnesses, including co-occurring disorders, in order to improve health and reduce illness, death, disability, and cost to society. In carrying out this mission it is necessary for service providers to collect sensitive items such as criminal justice involvement, use of alcohol or other drugs, as well as issues of mental health. The data that will be submitted by each grantee will be based in large part on data that most of the programs are already routinely collecting. This primarily includes data on client demographics, substance abuse and treatment history, services received, and client outcomes. These issues are essential to the service/treatment context. Grant projects use informed consent forms as required and as viewed appropriate by their individual organizations (see Attachment 5 for a Sample Consent Form); they use the appropriate forms for minor/adolescent participants requiring parental approval. Client/participant data are routinely collected and subject to the Federal Regulations on Human Subject Protection (45 CFR Part 46; OMB No. 0925-0404). Alcohol and drug abuse client/participant records in federally supported programs are also protected by 42 CFR Part 2.

12. Estimates of Annualized Hour Burden

A typical grantee currently collects intake, or pre-intervention information at the beginning of program contact, and many also collect standard discharge and follow-up information with similar items. Data are usually collected through interviews for the programs. Across all the SAMHSA discretionary services grants to which this application applies, it is estimated that these customary and usual business practices for services and treatment take about 20 minutes per data collection based on a pretest of the tool. Additional burden will only be created where grants are required to collect GPRA core measures at either intake, discharge, or follow-up points that are not customary and usual practices. In these cases the client’s/participant’s time and effort are required to gather additional information that would not have been part of normal treatment or service activities.

The total amount of time that is estimated for completion of core items based on the current programs in CSAP and CSAT is 20,170 hours (46,599 clients/participants in these programs complete 3 or 4, 20-minute data collections at specific time points and staff extract/upload data; note that for the 178,297 clients in one CSAT grant program there is no added burden see chart below.) That serves as the starting point for burden computations. CSAP estimates significant reduction in burden from the use of this GPRA tool, because of its transition to the NOMs tool, beginning in FY 2006. CSAP is not asking grantees funded in FY 2005 and previously to change instruments mid-stream, and is therefore requesting continued use of this instrument for those grantees funded prior to FY 2006, who have already begun the use of this instrument for data collection. This decision is based on the need for consistency of data pre, post, and follow-up.

The first value computed is the proportion of additional core GPRA items for a typical Center grant. This is done using the following formula:

Additional burden is calculated by multiplying this proportion times 20 minutes for each data collection (intake or baseline, discharge, 3-month follow-up, or 6-month follow-up). Informal testing showed that 20 minutes provided a conservative estimate for the maximum amount of time it would take for interviews.

Because the Centers have different types of grants and different core GPRA items, each Center has computed a specific Added Burden Proportion for its own grant programs.

• CSAP: There are a total of 32 adult items and 50 youth items (including record management) in the CSAP GPRA Client/Participant Outcome Measures for Discretionary Programs, which will take approximately 20 minutes per client/participant to administer at up to three data collection points (pre-intervention, post-intervention, 6-month follow-up or 12-month follow-up). Approximately 11 of the questions are routinely used in CSAP grant projects. The Added Burden Proportion is then (32-11)/32, or .66 for the adult items and (50-11)/50, or .78 for the youth, resulting in an average Added Burden Proportion of .72.

$ CSAT: There are 63 items (including record management) in the CSAT GPRA Client/Participant Outcome Measures for Discretionary Programs, which will take approximately 20 minutes per client/participant to administer at each of the three data collection points. However, 42 of the items are taken from the ASI, which is used in the substance abuse treatment field by researchers and providers as a baseline and follow-up instrument, or are considered standard items in the field. The resulting Added Burden Proportion is then (63-42)/63, or .33.

The following table shows the computation of added burden based on these proportions and the projected number of clients/participants completing the core GPRA items for each Center. An additional consideration of the data in the table is that this represents a maximum added burden. A final note is that some of the discretionary services programs may use a sampling approach for follow-up surveys. The number of clients/participants served by these discretionary services programs is fairly large and adequate sampling strategies are more cost effective and productive than a continuing census effort. Because the exact number of programs using sampling for follow up is not known, estimates were prepared using a census approach for all programs.

The calculation of maximum annual added burden hours and cost is as follows:

Estimates of Annualized Hour Burden ¹

Center/Form/ Respondent Type	Number of Respondents	Responses Per Respondent	Total Responses	Hours Per Response	Total Hour Burden	Added Burden Proportion2	Total Annual Burden Hours	Total Hour Cost / Respon-dent6
CSAP GPRA Participant Outcome Measures for Discretionary Programs
Participants
FY2007	7,000	3	21,000	.33	6,930	.72	4,990	$25,698
FY2008	3,000	3	9,000	.33	2,970	.72	2,138	$11,011
CSAP Subtotal	10,000	3	30,000	.33	9,900	.72	7,128	$36,709
CSAP Annualized Subtotal	5,000		15,000				3,564	$87,695
CSAT GPRA Client Outcome Measures for Discretionary Programs
Clients
Adults	28,000	3	84,000	.33	27,720	.33	9,148	$47,112
Adolescents	3,900	4	15,600	.33	5,148	.33	1,699	$8,750
Screening, Brief Intervention and Referral to Treatment (SBIRT) 3
Screening Only	150,618	1	150,618	.10	15,062	0	0
Brief Intervention	27,679	3	83,037	.16	13,286	0	0
Brief Tx & Referral to Tx	9,200	3	27,600	.33	9,108	.33	3,006	$15,481
SBIRT Client Subtotal	187,497		261,255		37,456		3,006	$15,481
Client Subtotal	219,397		360,855				13,853	$71,343
Data Extract by Grants4
Adult Records	400 grants	70 X 3	210	.16	34	--	34	$510
Adolescent Records	73 grants	53 X 4	212	.16	34	--	34	$510
Screening, Brief Intervention and Referral to Tx (SBIRT) Records
Screening Only	7 grants	21,517 X 1	21,517	.05	1,076	--	1,076	$16,140
Brief Intervention	7 grants	3,954 X 3	11,862	.08	949	--	949	$14,235
Brief Tx & Referral to Tx	7 grants	1,314 X 3	3,942	.16	631	--	631	$9,465
Data Extract Subtotal	480		37,743				2,724	$40,860
Upload5	5 grants		171,639	1 hr. per 6,000 records	29	--	29	$435
Upload Subtotal	5 grants		171,639				29	$435
CSAT Subtotal	219,896		570,237				16,606	$112,638
TOTAL	224,896		585,237				20,170	$200,333

NOTES:

1. This table represents the maximum additional burden if adult respondents provide three sets of responses/data and if CSAT adolescent respondents provide four sets of responses/data.

2. Added burden proportion is an adjustment reflecting customary and usual business practices programs engage in (e.g., they already collect the data items).

3. Screening, Brief Intervention, Treatment and Referral (SBIRT) grant program:

* 150,618 Screening Only (SO) respondents complete section A of the GPRA instrument, all of these items are asked during a customary and usual intake process resulting in zero burden; and

* 27,679 Brief Intervention (BI) respondents complete sections A & B of the GPRA instrument, all of these items are asked during a customary and usual intake process resulting in zero burden; and

* 9,200 Brief Treatment (BT) & Referral to Treatment (RT) respondents complete all sections of the GPRA instrument.

4. Data Extract by Grants: Grant burden for capturing customary and usual data.

5. Upload: 5 of the 7 SBIRT grants upload data; the other 2 grants conduct direct data entry.

6. Estimate based on $5.15 for program staff and $15 for IT staff.

The estimates in this table reflect the maximum annual burden for currently funded discretionary services programs. The number of clients/participants served in following years is estimated to be the same assuming level funding of the discretionary programs, resulting in the same annual burden estimate for those years.

13. Estimates of Annualized Cost Burden to Respondents

There are neither capital or startup costs nor are there any operation and maintenance costs.

14. Estimates of Annualized Cost to Government

The principal additional cost to the government for this project is the cost of a contract to collect the data from the various programs and to conduct analyses which generate routine reports from the data collected. The reports examine baseline characteristics as well as the changes between baseline, discharge, and each of the follow-up periods. It is the responsibility of the contractor to work with the Government Project Officer (GPO) when preparing reports that combine the client/participant services data with the annual reports of the project.

• CSAP: The estimated annualized cost for a contract for the GPRA mandate is $491,000 and the cost of 1 FTE staff responsible for the CSAP data collection effort is approximately the midpoint of a GS-14 range and is $103,594/year.

• CSAT: The estimated annualized cost for a contract for the GPRA mandate is $7.2 million and the cost of 1 FTE staff (25% for the midpoint of one GS-14 $25,899 and 75% for one GS-12 $48,786) responsible for the CSAT data collection effort is approximately $74,685/year.

15. Changes in Burden

Currently there are 24,960 hours in the OMB inventory. SAMHSA is requesting a total of 20,170 burden hours, an overall decrease of 4,790 hours. The program change is due to the following:

• CSAP—a reduction in burden of 5,097 hours (8,661 to 3,564) due to the phasing out of CSAP grantees and their participants using this tool based on year funded. Those funded prior to FY 2006 will continue to collect these data but the data collection for those funded in FY 2006 or after will use the NOMs tool (OMB No. 0930-0230).
  
  

• CSAT—an increase in burden of 2,906 hours (13,700 to 16,606) due to a program change that results from an increase in the number of persons to be served by the SBIRT grantees.
  
  

• CMHS—a reduction in burden of 2,599 hours as a result of a transfer from this data collection to its own separate OMB clearance.

16. Time Schedule

Data for the annual GPRA plan/report are needed by SAMHSA by September of each year. The discretionary services program data are readily available through the web-based system. Data are provided for the most recently completed calendar year to SAMHSA in May in order to assure analysis in time for the annual GPRA report. The annual GPRA report must be submitted to the U.S. Department of Health and Human Services (the Department) and to OMB by September and is included in the President's annual budget request which is released to the public February 1st. Data may be refined and added to the final Presidential budget request after the Department submits its initial GPRA report.

Analysis/Publication Plans

Client/participant outcome data will be collected through the web site. Data will be used to report to Congress regarding the GPRA as specified in the SAMHSA Annual Justifications of Budget Estimates. The data might also be used for specific comparisons relative to ONDCP National Drug Control Strategic Goals, especially for some of the secondary treatment outcomes (e.g., homelessness).

In the future, the indicators for clients/participants served under these programs might be compared to similar indicators for clients/participants served under block grant programs as a general indicator of whether the programs are doing better than "typical" services. This could be done for discretionary services programs as a group or for specific programs.

SAMHSA and each of its Centers specifically will use the data for annual reporting required by GPRA on the previously stated items, comparing baseline with discharge and follow-up data. The GPRA dataset will consist of each element coded into the reporting categories as seen in Attachment 1. These data are at the client/participant record level. The SAMHSA GPRA client/participant outcome data will be aggregated at the following levels: Project/Grantee, Program/Division, and Activity. The analysis will be organized around SAMHSA's GPRA measures ─ the NOMs.

Baseline level analysis involves using frequency distributions and measures of central tendency to describe the populations across the GPRA client/participant outcomes and by various demographic groups (e.g., gender, race, ethnicity, age, and level of education). The client/participant will be followed longitudinally and the GPRA client/participant outcome items will be re-administered again at discharge and 6 months after baseline. The follow-up data also will be described using frequency distributions and measures of central tendency. Change will be addressed by comparing the discharge and follow-up measurements with baseline data for each client/participant. The percent of clients/participants showing the target changes will be calculated on each of the GPRA client/participant outcome measures that are categorical (as referenced in Section A.1 and shown in Attachment 3). For continuous items, mean differences will be calculated. Tables will be constructed to describe the change across projects on client/participant outcomes.

It is important to note that each Center is responsible for its own analyses of the data. Common analyses will be used as appropriate for GPRA purposes, but control of the data rests with the Center funding the grant. The Centers submit a GPRA report to SAMHSA Office of the Administrator and SAMHSA then synthesizes results from the Centers in a descriptive manner for the GPRA report.

There also will be Center unique analysis of these data because each Center has a distinct set of programs. The data items collected will be analyzed and presented in GPRA reports using basic descriptive statistics. On the principal outcome items (e.g., drug use, criminal involvement, and employment), the proportion of individuals showing improvement from baseline to discharge and follow-up (baseline to discharge, baseline to 3 months, baseline to 6 months) will be calculated and aggregated at the program level (e.g., discretionary services). If deemed necessary for CSAT specific issues, the data will be examined at the individual activity level (e.g., Addiction Treatment for Homeless or Treatment Drug Court). Occasionally, the results will be examined for subpopulations of interest within individual activities (e.g., by age or by gender).

See Attachment 6 for sample table shells that are generated from available reports.

17. Display of Expiration Date

The expiration date for OMB approval will be displayed on all data collection instruments for which approval is being sought.

18. Exceptions to Certification Statement

This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions. The certifications are included in this submission.

B. STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

The Center for Substance Abuse Prevention and the Center for Substance Abuse Treatment will all utilize the following strategies:

• CSAP: In the case of the State Incentive Grant (SIG) program, each state will sample a minimum of one program in each of three (out of five) domains: Individual/ Peer, Family, School, Community, Environmental. While each state may select which of the domains to target, CSAP expects sufficient and adequate sampling to provide adequate measures of performance at various levels of aggregation. If a program does not have a sufficient number of participants, additional programs will be added until a sufficient number is achieved per domain. A census approach is used for the participants within those programs sampled.

• CSAT: The CSAT estimated universe of individual respondents is 31,900 individuals per year across all the discretionary grant programs included in the request. For all the programs included, the starting point for use of the measures is a census of all clients/participants at intake (or initial contact). Beyond the initial census, there will be considerable variation in the number of respondents receiving the core measures since there is variability in the measurement strategy used for each program. The options include no follow-up or sampling follow-up for some discretionary services programs. Grantees will be directed to achieve a minimum of an 80 percent response rate for all clients/participants targeted for follow-up efforts. This guidance will apply to the response rate as defined against the intake or baseline census. The sampling strategy for the GPRA measures will be specific to CSAT and the capabilities of the programs under review.

CSAT’s strategy for conducting follow-up is presented here. For CSAT’s discretionary services grantee portfolio, the strategy for baseline, discharge, and follow-up will either be a complete census or a sample, depending on the number of clients the program plans to serve, the amount of variation expected, and the local project activities. The guidance to be given to grantees in this area will be based on project minimum cell size for each analysis of interest. SAMHSA will provide guidance for deciding whether to census or sample for follow-up as part of its instructions to grantees. A sampling rate will be recommended based on the anticipated program population. For example, if it is determined in further analysis of GPRA performance plans that a minimum cell size of 40 is needed for appropriate sensitivity to examine performance on a given measure, a sampling rate will be recommended so that 1 of every x clients/participants will be followed-up to achieve a sample size of at least 40 for each appropriate cell. Where it is not clear what the number of participants will be, an alternate strategy of following up the first 40 in each cell of interest would be recommended, with a sampling strategy initiated once the minimum was reached. Later adjustments could be made if a larger number of participants were actually served. This approach will reduce the actual burden because the total number of clients followed-up will be less than the total number collected at baseline.

2. Information Collection Procedures

Information collection procedures will vary by type of program. The client/participant outcome measures for most providers will be extracted from previously established databases. Intake/baseline information is obtained by intake workers and/or counselors. For clients/participants still in treatment 6 months later, the information will be obtained in the same way. In instances where clients/participants are no longer in direct contact with the service provider, staff from the program will locate the clients/participants and conduct the follow-up interviews. These interviews are to be conducted face-to-face but can be conducted over the phone in extenuating circumstances and with permission of the Government Project Officer.

Some programs collect their client/participant information using paper and pencil methods. This project will not interfere with ongoing program operations. Programs will submit their data electronically via a web-based data entry process or upload process. The data for those clients/participants with baseline, discharge, and follow-up data are matched using a unique encrypted client/participant identifier.

3. Methods to Maximize Response Rates

Each grantee will have established its own client/participant follow-up procedures as part of the original protocol. At the time of intake, information is typically obtained from clients/

participants to assist with locating them later. This includes information on current residents plus information on one or two other individuals who are likely to know where they are if they have re-located. In addition, some providers are adept at using other community resources to assist with locating clients/participants. Clients/participants are typically quite cooperative with provider staff because of the relationship established during treatment. Since all participating grant programs propose a census at initial intake, considerable options also exist for non-respondent analysis and associated adjustments to the data such as weighting.

Follow-up has been a challenge to some grantees given the remote locations that they serve and the challenge of locating clients/participants as far out as 6 months. For grantees that have not been aware of the strategies they can employ to begin the follow-up process at intake, how to maintain contact with clients/participants, and the importance of good locator forms, several strategies have been implemented to assist the grantees with followup. First, follow-up training is offered which assists grantees in learning about and conducting follow-up at their sites. This program is offered to all grantees and after the grantees are trained through the grantee orientation process, monthly follow-up trainings are offered for those that need additional training or for new project staff. Individual grantee technical assistance is also available for sites that need additional follow-up instruction. These group and individual trainings are conducted by follow-up experts. Each grantee receives a follow-up tracking manual at these trainings that may be used as a future reference. A second strategy provides the grantees with data status reports on how close they are to meeting their follow-up goals. These reports are available from the web-based system to the grantees and Government Project Officers for the grants they are responsible. A third strategy is the automatic, system generated notice of when follow-up interviews are due for each client/participant. A fourth strategy provides technical assistance at national meetings. Experts, including grantees, have been identified and asked to make presentations at national grantee meetings on how to conduct follow-up. These sessions are well attended by grantees. It is anticipated that these strategies will continue to improve the follow-up rates and it is continually stressed to the grantees that a minimum 80 percent follow-up rate is expected.

4. Tests of Procedures

Most of the data elements in the data sets have been taken from established data collection instruments that have a long history of use in the substance abuse field and have already been tested for validity and reliability, (i.e., ASI).

Feedback from the grantees also indicates that they routinely collect the same information requested of this data collection tool and some have integrated this tool into other tools that they routinely use to gather information. Some grantees report that they collect information in greater detail, (i.e., more response alternatives), but these are collapsed into standard categories.

5. Statistical Consultants

Responsible individuals for each of the Centers are:

• CSAP: Beverlie Fallik, Ph.D. 240-276-2438

• CSAT: Deepa Avula 240-276-2960

Patrice Roth 240-276-2973

The individual responsible for statistical consultation of this data collection is: Sanjeev Sridharan, Ph.D., Westat at 301-251-1500.

ATTACHMENTS

Attachment 1: CSAT GPRA Client Outcome Measures for Discretionary Programs and Instructions

Attachment 2: CSAP GPRA Participant Outcome Measures for Discretionary Programs and Instructions

Attachment 3: SAMHSA GPRA Client/Participant Outcome Measures for Discretionary Programs Matrix

Attachment 4: CSAT GPRA Web-based Data Entry Screens

Attachment 5: Sample Consent Form

Attachment 6: CSAT Sample Reports

17