Justification

Center for Mental Health Services and Center for Substance Abuse Treatment

Co-Occurring Disorders Integration and Innovation (CODI)

Customer Satisfaction Survey

1. Background

The Substance Abuse and Mental Health Services Administration (SAMHSA) was established under the Alcohol, Drug Abuse, and Mental Health Administration Reorganization Act (Public Law No. 102-321) as an operating division of the Department of Health and Human Services (HHS). SAMHSA’s mission is to provide national leadership in the prevention and treatment of addictive and mental disorders, through appropriate programs and services for individuals who suffer from mental health and/or substance abuse disorders. Consistent with its mission, two centers within SAMHSA, the Center for Mental Health Services (CMHS) and the Center for Substance Abuse Treatment (CSAT) collaboratively established Co-Occurring Disorders Integration and Innovation (CODI) to address the needs of people who are challenged with both mental health disorders and substance abuse problems.

Both CMHS and CSAT work at the federal level to carry out their respective missions. Within SAMHSA, CMHS promotes the prevention and treatment of mental disorders by identifying national mental health goals and developing strategies to meet them. CMHS works to improve the quality of programs that serve both individuals suffering from these disorders and their families. CSAT provides national leadership in developing and administering programs focusing on the treatment of substance abuse. CSAT carries out its work with states, local communities, and health care providers by providing financial assistance to improve and expand programs for treating substance abuse. Both CMHS and CSAT administer and evaluate grants and programs that help states and local governments provide mental health care and addictive treatment services, respectively.

CMHS’ and CSAT’s collaborative efforts around CODI focus on a common goal, which is to facilitate the adoption of evidence-based practices (EBPs) in mental health and substance abuse treatment services as well as their integration into primary care settings. Knowledge transfer and information dissemination activities are, central to the CODI and are aimed at service systems, providers and consumers. These activities are intended to promote improvements in practice, service delivery, and service outcomes for consumers. CODI uses three primary mechanisms to support its knowledge transfer and dissemination efforts, which include a central website, product development, and technical assistance activities. Guided by the CODI Knowledge Transfer Approach (KTA) Plan, all CODI efforts are organized around six priority areas that focus on 1) Systems and Services Integration; 2) Screening and Assessment, 3) Workforce Development; 4) EBP Implementation and Clinical Training; 5) Financing; and 6) Collection and Use of Data. Collectively, these activities are intended to promote access to effective integrated services for persons with co-occurring disorders.

2. Product/Activity to be Assessed

Through the administration of specific satisfaction survey instruments CODI will collect the following information.

1. All products developed under the CODI will undergo testing to ensure that CODI products meet the needs of its audiences (see Formative Product Assessment Survey in Appendix A);

2. To assess whether target audiences are satisfied with CODI technical assistance, and find these CODI efforts to be appropriate, a Survey of Satisfaction with CODI Events will be administered (see Survey of Satisfaction with CODI Events in Appendix B).

3. To assess the satisfaction with CODI technical assistance over time on members of the CODI Learning Community Networks, the Impact Survey of CODI Learning Communities will be administered (see Impact Survey of CODI Learning Communities in Appendix C).

C. Statement of Objectives

1. Product Development – The objective of the product development data collection activities are intended to assess user perspectives and satisfaction with CODI products in order to improve their usability. Administration of the Formative Product Assessment Survey will assist with this assessment. These questions seek the opinions of users about a product’s content, layout, and readability, perceived usefulness. The survey will be administered either in print form in-person or as emailed request for survey response that will link to a web-administered version of the survey. Actual administration of the survey will be based on the type of product being evaluated. For example, for print products, Intercept Testing may be used in which target audiences are recruited during conferences and workshops. This method requests randomly selected participants from event audiences to review products-in progress and respond to survey questions.

2. CODI Events – CODI events include CODI-sponsored grantee meetings, remote technical assistance (webinars and online training), and in-person conference presentations or workshops. The objective of data collection activities around CODI events is to assess participants’ satisfaction with the training content and its delivery during an event. The Survey of Satisfaction with CODI Events inquires about the organization, presentation of information, products received as part of the event, quality of the products, plans to share the products, usefulness of the event, and participants’ interest in having other event topics covered in training events and products developed. Product related questions will be included for Events that include introduction to CODI products.

3. Impact of CODI’s Learning Communities (CODI-LCs) -The objective of data collection activities around the CODI Learning Communities is to measure satisfaction and goal attainment by participants across the six CODI priority areas addressed in the learning community events. Impact Survey of CODI Learning Communities will be administered to Learning Community members 6 – 9 months following their first CODI event. During the 6 – 9 months between initiation into the Learning Community and the administration of the Impact Survey, it is expected that Learning Community members will have been offered the opportunity to participate in approximately one CODI event each month.

Respondents and Timeline - Table 1, which follows, identifies the respondents and frequency for administering the survey instruments per year. Generally, data collection activities are ongoing throughout the duration of the CODI contract and are linked to specific events or activities.

Table 1. Respondents and Method Survey Administration by Survey Instrument
Data Collection Instrument	Respondent(s)	Method of Survey Administration	Frequency
Formative Product Assessment Survey	CODI target audiences and sub-populations	Paper and Web-based Survey	Throughout the CODI contract as the product development process requires
Survey of Satisfaction with CODI Events	Direct participants of CODI events	Paper and Web-based Survey	Once per CODI event; Remote TA (webinars and online trainings), averaging 150 per event, 12 times per year. CODI involvement in conference presentations, grantee meetings, and in person workshops, 50 per event, 7 times per year. COSIG Grantee Meetings, 60 per event, 1 time per year.
Impact Survey of CODI -LCs	Direct participants in Learning Communities	Web-based Survey	Once at 6 months after beginning participation in the in Learning Communities

Sample Size and Rationale - Table 2 below summarizes the sample size and rationale for sampling estimates for administering the surveys to the intended audiences.

Table 2. Sample Size and Rationale for Sampling Estimates by Type of Survey
Data Collection Instrument	Sample Size	Rationale for Sampling Estimates
Formative Product Assessment Survey	Conference attendee intercepts and emails: 600 surveys total per year. N will vary by product type.	600 surveys represents an upper limit of audience contacts distributed among product assessments. Audience data collection for formative research is not intended to be statistically projectable but is meant to identify potential problems with product quality.
Survey of Satisfaction with CODI Events	All participants; Remote TA (webinars and online trainings) averaging 150 per event, 12 times/year. CODI involvement in conference presentations, grantee meetings, and in person workshops, 50 /event, 7 times/ year. COSIG Grantee Meetings, 60 /event, 1 time/year.	No sampling; all participants are to be surveyed.
Impact Survey of CODI Learning Communities	7 Learning Communities per year with 25 members each; survey of 1 person per community; up to 175 surveys annually.	No sampling; each participating community is to be surveyed.

Response Rate

Based on past experience of collecting similar data using paper-based and web-based surveys, SAMHSA anticipates receiving responses from approximately 85% of the participants.

Privacy Protection

The privacy of respondents will be protected to the extent of the law in the reporting of the survey data. When respondents complete the online survey, their responses will be transmitted directly to Cloudburst. Upon processing by Cloudburst, the responses will be included in a data file stored in a secured portal on the CODI website. This portal is only accessible by CODI staff, and access to the area where the data file is stored is password protected. In addition to these precautions, respondents will be assured that results of this survey will only be presented in aggregate form so that individual responses cannot be identified. Similarly, respondents completing paper-based surveys will do so anonymously and reporting of data from all paper-based surveys will be aggregated prior to reporting. Paper surveys will be collected by Cloudburst, scanned into an archive, and data from the paper copies will be entered into a database application by Cloudburst for reporting of basic descriptive statistics. Paper copies will be retained in a secure file in Cloudburst’s Landover, MD offices.

D. Annual Response Burden Estimate

This data collection effort will not involve start-up or operational/maintenance costs to respondents. The maximum number of responses has been estimated for each potential contact with the respondents. (1) The average burden per response was estimated from a pilot test of the survey. On average, it took approximately 5 minutes (.08 hour) to complete the product survey, 4 minutes (.07 hour) to complete an event survey, and 15 minutes for an impact survey. (2) The hourly wage estimate for survey participants was drawn from a figure provided in a sample Independent Government Cost Estimate by the SAMHSA Division of Contracts Management on the SAMHSA intranet.

Table 3: Estimated annualized burden hours for respondents
Instrument	Number of Respondents	Average Number of Responses per Respondent	Average Burden per Response (hrs)	Estimated Total Average Respondent Burden Hours
Formative Product Assessment Survey	600	1	.08	50.0
Survey of Satisfaction with CODI Events	2210	1	.07	147.3
Impact Survey of CODI Learning Communities	175	1	.25	43.8
Total	2,985			241.1

Table 4 shows the estimated annualized cost burden for the respondents. The Bureau of Labor Statistics reported that the average hourly wage for "mental health and substance abuse social workers in the United States was $19.88 in May 2009. An estimate of $20 per hour allows for inflation and represents a conservative estimate of the wages of the respondents. Therefore, the total estimated cost burden for respondents is $4,822, based on the total estimated annualized burden hours of 241.1 hours.

Table 4: Estimated annualized cost burden for respondents
Estimated Total Average Respondent Burden Hours	Hourly Wage Cost*	Total Hour Cost ($)
50.0	$20	$1,000
147.3	$20	$2,947
43.8	$20	$875
241.1	$20	$4,822

*Based upon the average wages, “Occupational Employment and Wages, May 2009

21-1023 Mental Health and Substance Abuse Social Workers in the United States, May 2009,” U.S. Department of Labor, Bureau of Labor Statistics.

E. Methods Used to Develop and Test the Survey Questions

Research staff at Westat and Cloudburst Consulting Group in collaboration with CMHS and CSAT Government Task Order Officers developed the proposed customer satisfaction surveys. The surveys were timed in an online environment among the evaluation workgroup and were also pretested with a small group of attendees at an event and revised based on their feedback. The average amount of time reported to complete a survey was 4 minutes. Westat and Cloudburst Consulting Group, the organizations responsible for conducting the survey, have extensive experience in survey design and have used similar types of questions in the past (both in paper-based surveys and web-based surveys).

F. Federal Task Order Officers

Onaje Salim, MA, LPC

Public Health Advisor

SAMHSA

Center for Substance Abuse Treatment

Co-Occurring and Homeless Activities Branch

1 Choke Cherry Road, Room 5-1056

Rockville, MD  20857

Phone: 240-276-2895

Fax: 240-276-2800

Email: [email protected]

Dr. Charlene LeFauve

CODI Task Order Officer

Center for Substance Abuse Treatment

Substance Abuse and Mental Health Services Administration

Choke Cherry Road

Rockville, MD 20857

Phone: 240-276-2787

Cellular: 301-367-9580

Fax: 240-276-2800

Email: [email protected]

Deborah Stone, Ph.D.

SAMHSA/Center for Mental Health Services

DSSI/Homeless Programs Branch

1 Choke Cherry Road, Room 6-1031

Rockville, MD 20850

240-276-2411

FAX: 240-276-1930

E-mail: [email protected]

8