Justification

OMB Customer Satisfaction Survey Clearance Request

For the Center for Mental Health Services

CMHS-GPRA Satisfaction of Technical Assistance Survey

1. Product/Activity to be Assessed

Under the Government Performance and Results Act (GPRA), the Substance Abuse and Mental Health Services Administration (SAMHSA) Center for Mental Health Services (CMHS) is responsible for monitoring program performance across all of its budgetary lines. The CMHS-TRansformation ACcountability (CMHS-TRAC) system consolidates GPRA reporting and related activities for CMHS’ entire discretionary program of Consumer Services, Technical Assistance, Prevention, and Infrastructure Development grantees under one umbrella. Specifically, TRAC provides support for accountability and performance assessments of these programs and makes such information available to CMHS Project Officers for improved project oversight and to CMHS management for improved staff and grantee portfolio oversight.

TRAC is a web-based data system for the programs mentioned above. The data collection, known as the as the “Consumer Services National Outcome Measures”, is approved under OMB No. 0930-0285 (Expiration Date 04/30/2010).

CMHS will conduct this online customer satisfaction survey, named Survey of Satisfaction with CMHS Technical Assistance of grantee Project Directors (PDs) that were provided services by a CMHS-funded Technical Assistance grantee or contractor. The purpose of this data collection is to quantify users’ satisfaction with the Technical Assistance (TA) services provided, assess whether services provided met a minimum level of competency, and determine the extent to which TA had an impact on grantees’ ability to conduct the grant. Grantee PDs will be assessed annually for the duration of their grant; they will be given one month to complete the online survey.

B. Statement of Objectives

Results of this survey of TA customer satisfaction and competencies, the Survey of Satisfaction with CMHS Technical Assistance, will be used by SAMHSA/CMHS to assess grantee PDs’ opinions regarding the usefulness of and their satisfaction with the technical assistance services provided to their grants. Findings from this survey will allow SAMHSA/CMHS to better manage their TA Providers and to inform those TA Providers of possible improvements in delivering technical assistance.

The survey will assess the following components of the CMHS Technical Assistance Services and providers using both close-ended quantitative items and open-ended qualitative items (see Attachment A).

• Receipt of, Need for, and Satisfaction with Technical Assistance by Type – These questions assess whether grantee’s received, needed, and if received, were satisfied with the technical assistance by type. The types of Technical Assistance identified are: Regular; Ongoing Consultation/Discussion; Customized Technical Assistance (TA); Trainings, Webinars, and Other Events; Annual or Semi-Annual Grantee Meetings; Resource Materials; Information or Support via the TA Provider’s Website; and Site Visits.

• Receipt of and Contribution of Technical Assistance to Grantee’s Performance Technical Assistance by Domain – These questions assess whether grantee’s received, needed, and if received, were satisfied with the technical assistance by Domain. The Domains of Technical Assistance identified are: Decreasing Criminal Justice Involvement; Improving Access to Services or Service Capacity; Improving Client Perception of Care; Improving Cost Effectiveness; Improving Consumer Functioning; Increasing Consumer Social Supports/Social Connectedness; Increasing or Retaining Consumer Employment or Returning to/Staying in School; Increasing Stability in Housing; Increasing the Use of Evidence-Based Practices; and Reducing Consumer Utilization of Psychiatric Hospitalization.

• Receipt of and Contribution of Technical Assistance to Grantee’s Performance by Topic – These questions assess whether grantee’s received, needed, and if received, were satisfied with the technical assistance by Topic. The Topics of Technical Assistance identified are: Addressing Co-occurring Mental Health and Substance Use Disorders; Building and Maintaining Coalitions; Conducting Project Evaluation; Developing or Implementing Cultural Competence/Appropriateness; Developing or Implementing Communications and Social Marketing; Developing Sustainability Plans; Financing; Identifying/Selecting Best Practices Programs; Implementing Best Practices Programs; Implementing Sustainability Plans; Involving Consumer, Family, & Youth in Policy, Programs and Evaluation; Making Services Consumer-, Family-, & Youth-Driven; Needs Assessment; Project Management; Strategic Planning; and Workforce Development.

• Core Technical Assistance Competencies – These questions will assess the performance of TA Providers regarding core service competencies.

• Knowledge, Skills, and Outcomes of Technical Assistance – These questions will provide information on the grantee’s opinions regarding the attainment or enhancement of knowledge and skills as a result of the TA provided and whether the TA helped to achieve programmatic outcomes.

• General Feedback and Comments: These open-ended questions query grantees’ regarding the need for additional TA not listed on the survey and provide free-text fields for general comments.

C. Methodology

Data Collection Method

This customer satisfaction survey will be conducted using a web-based survey. The annual process will include sending an email reminder to grantee PDs two weeks prior to the beginning of the data collection period to notify them of this impending GPRA data collection request (see Attachment B). Additionally, instructions regarding the due date for completion of the survey and a link (see Attachment C) to the survey will be provided. Once data collection is initiated users will be prompted to complete the survey every time they log into the CMHS-TRAC system. The grantee PDs will have one month to submit the information online. After a user completes the survey they will no longer receive the prompt when they log into the CMHS-TRAC system.

Respondents

Respondents will include all CMHS Grantee PDs with the exception of CMHS Technical Assistance Providers. All PDs who log into the CMHS-TRAC system during this time will be prompted to participate in the survey; participation in the survey is strongly encouraged though not required. This online survey will be conducted on an annual basis every October, to enable the information to be used in the assessment of goals and performance for the next Federal Fiscal Year.

Sample Size and Rationale

All PDs for grants that were provided services by a CMHS-funded Technical Assistance Provider will be requested to participate in the survey. While some TA Providers serve the general public, the rationale for choosing to survey only grantee PDs is currently the CMHS-TRAC system lacks the means to track every user of CMHS-funded Technical Assistance services and thus is unable to receive their feedback.

There are approximately 800 CMHS-funded Grantees which may be receiving service by a Technical Assistance Service Provider. Although this is not a required survey, we are using 800 as the estimated maximum number of respondents.

Timeline of Data Collection

The online survey will be conducted during the month of October 2009, for the initial round, and annually thereafter. This timeframe was chosen to allow users an adequate amount of time to complete the survey. It also allows for the data to be aggregated and incorporated into funding decisions as needed.

Response Rate

Based on past experience of collecting similar data using web-based surveys, SAMHSA anticipates receiving responses from approximately 80-85% of the PDs. In order to maximize the number of respondents, users will be prompted to complete the survey every time they log into the CMHS-TRAC system. After a user completes the survey, they will no longer receive the prompt. The survey prompt will change after 24 days of data collection have passed. This follow-up prompt will remind users about the survey and restate that their participation is strongly encouraged (see Attachment D).

Additionally, since performance measures are currently being developed for some of CMHS’ Grantees, those respondents will not be able to be notified via the CMHS-TRAC system. All Grant PDs will be sent emails notifying them of the data collection requirement two weeks in advance and when the data collection period is initiated; one reminder will be sent to all non-responders during the last week of the data collection period (Attachments B - D).

Confidentiality

The identity of respondents will be protected in the reporting of the online survey data. When respondents complete the online survey, their responses will be transmitted directly to a data file stored on Westat’s server. This server is only accessible by TRAC staff, and access to the area where the data file is stored is password protected. In addition to these precautions, respondents will be assured that results of this survey will only be presented in aggregate form so that individual responses cannot be identified.

D. Annual Response Burden Estimate

This data collection effort will not involve start-up or operational/maintenance costs to respondents. The maximum number of responses has been estimated for each potential contact with the respondents.

Survey	Number of Respondents(a)	Responses per Respondent	Hours per Respondent(b)	Annual Burden Hours	Hourly Wage(c)	Total Hourly Cost
Web-based Customer Satisfaction Survey	800	1	.16	128	$31.00	$3,968

1. The number of responses has been based on information from the Task Order Officer and CMHS Government Project Officers.

2. The average burden per response was estimated from a pilot test of the survey. On average, it took approximately 10 minutes (.16 hour) to complete the survey.

3. The hourly wage estimate for CMHS grant Project Directors was drawn from a figure provided in a sample Independent Government Cost Estimate by the SAMHSA Division of Contracts Management on the SAMHSA intranet.

E. Methods Used to Develop and Test the Survey Questions

The proposed customer satisfaction survey was developed by research staff at Westat in collaboration with CMHS Government Project Officers and select Cross-site Evaluators. The MN SMART‛s Relationship Based Professional Development Strategy Specific Competencies, Competencies for Effective Technical Assistance (2007), developed by Concordia University was used as a model for question formulation (www.mnsmart.org). Scott Formica of Social Science Research Evaluation, Inc. (SSRE) was also consulted regarding SSRE’s evaluation of the Safe Schools Healthy Students Technical Assistance Program. Multiple members of the TRAC team, CMHS staff, and grantees reviewed and provided input for initial drafts of this redesigned survey. The survey was also thoroughly tested with grantees and revised based on their feedback; three separate pilot studies were conducted with grantees selected by CMHS staff. The average amount of time reported to complete the survey was 10 minutes. Westat, the organization responsible for conducting the survey, has extensive experience in survey design and has used similar types of questions in the past (both in paper-based surveys and web-based surveys).

F. Federal Project Officers

Dr. Crystal Blyler

TRAC Project Officer

SAMHSA/CMHS

1 Choke Cherry Road

Rockville, MD 20857

Phone: 240-276-1910

Fax: 240-276-1910

Email: [email protected]

List of Attachments

Attachment A – TRAC Online Users’ Survey

Attachment B—Technical Assistance GPRA Data Collection Request Notification

Attachment C – Initial Prompt for TRAC Online Users’ Survey

Attachment D – Follow-Up Prompt for TRAC Online Users’ Survey

Attachment A – TRAC Online Users’ Survey

Attachment B—Technical Assistance GPRA Data Collection Requested Notification

CMHS Grant Project Director,

Each year (beginning in 2009), the SAMHSA Center for Mental Health Services (CMHS) conducts a survey of its grantees to determine their level of satisfaction with the technical assistance (TA) they have received over the past year from CMHS TA providers in connection with the grant program. The Project Director for each CMHS grant is requested to complete the survey within a specified time period.

This notice is to inform you that within the next two weeks you will be receiving instructions to complete the survey regarding the TA you have received from CMHS TA providers in connection with your grant over the past year. If you have questions about this request, please contact the CMHS Project Officer for your grant, or the CMHS Project Officer for the survey ([email protected], 240-276-1910). CMHS appreciates your honest responses to the survey and will use the data to improve TA services.

Attachment C – Initial Prompt for TRAC Online Users’ Survey

CMHS Grant Project Director, it’s time to complete the annual Survey of Satisfaction with CMHS Technical Assistance (TA). This survey is important to CMHS in monitoring the quality of its TA programs and ensuring Government Reporting Performance Act (GPRA) requirements are met. Completion of this survey is strongly encouraged for all CMHS Grant Project Directors whose grant received technical assistance within the past year. Your identity will be protected in the reporting of the online survey data. Your response should indicate your satisfaction of the TA received from your assigned CMHS TA Provider.

The survey is available online at: https://. Please complete it by XX/XX/XXXX. Contact the TRAC Help Desk if you need any further assistance ([email protected] or (888) 219-0238).

Attachment D – Follow-Up Prompt for TRAC Online Users’ Survey

CMHS Grant Project Director, according to our records you have not yet completed the Survey of Satisfaction with CMHS Technical Assistance (TA) for your grant. This survey is important to CMHS in monitoring the quality of its TA programs and ensuring Government Reporting Performance Act (GPRA) requirements are met. All Grant Project Directors are requested to complete this survey annually. Your identity will be protected in the reporting of the online survey data. Your response should indicate your satisfaction of the TA received from your assigned CMHS TA Provider.

 

The survey is available online at: https://. Please complete this survey by XX/XX/XXXX. Contact the TRAC Help Desk if you need any further assistance ([email protected] or (888) 219-0238).

5