Justification

OMB Customer Satisfaction Survey Clearance Request

For Shared Decision-Making in Mental Health Decision Support Tools

A. Product/Activity to be assessed

The purpose of the assessment is to evaluate the extent of user satisfaction with three decision- support tools, developed for use by consumers of mental health services and mental health service providers, as described below.

Product #1: The first tool is a workbook entitled, “What Is Right For Me? How to Make Decisions in Everyday Life,” (see attachment #1) designed to support the process of decision-making by consumers of mental health services. This workbook was developed to help individuals make a decision they find difficult. The workbook teaches basic skills of decision-making, and includes step-by-step guidance and worksheets for making a difficult decision as well as tips for talking to others – including providers – about the decision.

Product #2: The second tool is a companion workbook to product #1 entitled, “Supporting Choice: Helping Others Make Important Decisions,” (see attachment #2) developed for helpers in the decision-making process, including mental health service providers. This workbook was designed to be used in conjunction with the product #1.

Product #3: The third tool is an interactive, computer-based Decision Aid about Anti-psychotic Medications (see access information below) that offers objective information about medications, side effects, and a variety of treatment and service options, including complementary and alternative approaches and recovery activities. The decision aid was constructed to help consumers of mental health services make an informed choice about a mental health care decision, including enabling them to prepare and print a personal report that can be discussed with mental health service providers or kept for personal use.

The development of these tools is being funded under a contract from the federal Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Mental Health Services (CMHS) to Advocates for Human Potential, Inc. (AHP). The development and evaluation of the tools is being conducted by AHP.

Product #3 Access Information

Weblink: http://demo3.westat.com/graphics/SDM_DA_Working/index.html

Username: sdm
Password: Graph2010!

This application for OMB approval is specifically to conduct surveys for assessing the satisfaction of consumers of mental health services and mental health service providers with the three products described above. There are four related but separate survey instruments as follows:

Survey #1: What Is Right For Me Workbook User Satisfaction Survey for Consumers of Mental Health Services (attachment #3); this survey intends to measure the satisfaction with product #1 from the consumer perspective.

Survey #2: Decision Support Workbook User Satisfaction Survey for Mental Health Service Providers (attachment #4); this survey intends to measure the satisfaction with product #1 and product #2 from the provider perspective.

Survey #3: Decision Aid on Antipsychotic Medications User Satisfaction Survey for Consumers of Mental Health Services (attachment #5); this survey intends to measure satisfaction with product #3 from the consumer perspective.

Survey #4: Decision Aid on Antipsychotic Medications User Satisfaction Survey for Mental Health Service Providers (attachment #6); this survey intends to measure satisfaction with product #3 from the provider perspective.

These surveys, which will be group administered in a paper and pencil format, are included in this application. The surveys will be conducted during 2010.

B. Objectives of proposed customer satisfaction effort

The objective of the user satisfaction surveys is to assess the extent to which consumers and providers are satisfied with the usefulness of the products in supporting decision-making by consumers, including shared decision-making with providers about treatment and services. The results of the surveys will be used by SAMHSA to determine whether to promote use of these decision support tools to mental health care consumers and providers.

Survey #1: The What Is Right For Me Workbook User Satisfaction Survey for Consumers of Mental Health Services will capture information about product #1, including:

• Overall impressions of the format, organization, and content of product #1;

• Perceived usefulness of product #1 in supporting decision-making generally;

• Perceived usefulness of the product #1 in developing skills and building confidence about decision-making;

• Perceived usefulness of the product #1 in supporting shared decision-making about mental health services;

• Characteristics of mental health services consumers using product #1;

• Open comments.

Survey #2: The Decision Support Workbooks User Satisfaction Survey for Mental Health Service Providers will capture information about products #1 and #2, including:

• Perceived usefulness of product #1 in supporting decision-making generally;

• Perceived usefulness of the product #1 in supporting shared decision-making about mental health services;

• Overall impressions of the format, organization, and content of product #2;

• Perceived utility of product #2 in supporting decision-making;

• Provider training and role;

• Open comments.

Survey #3: The Decision Aid on Antipsychotic Medications User Satisfaction Survey for Consumers of Mental Health Services will capture information about product #3, to include the following:

• Overall impressions of the format, organization, and content of product #3;

• Perceived utility of product #3 in increasing knowledge about antipsychotic medications and alternative and complementary approaches;

• Perceived utility of product #3 in supporting decisions about antipsychotic medications and other mental health treatment or services;

• Perceived usefulness of product #3 in supporting shared decision-making about mental health treatment or services;

• Characteristics of mental health services consumers using product #3;

• Open comments.

Survey Instrument #4: The Decision Aid on Antipsychotic Medications User Satisfaction Survey for Mental Health Service Providers will capture information about product #3, to include the following:

• Perceived utility of product #3 in supporting decisions about antipsychotic medications and other mental health treatment or services;

• Perceived usefulness of product #3 in supporting shared decision-making about mental health treatment or services;

• Provider training and role;

• Open comments.

C. Overview of data collection methodology

Satisfaction Survey Testing Sites

Three testing sites for administration of the four user satisfaction surveys will be selected. The testing sites will be mental health agencies that provide a range of different mental health programs and services. Targeted programs at these three sites will include outpatient mental health services, psychosocial rehabilitation programs, peer support programs, and pharmacological treatment services. Tentative sites for testing are as follows:

1. Riverside Community Care, 450 Washington Street, Dedham, MA  02026

2.  Howard Center for Human Services, 208 Flynn Avenue, Burlington, VT 05401

3.  VA Medical Center West Haven, 950 Campbell Avenue, West Haven, CT 06516

Testing of the three products will be conducted in two phases. The first phase will test product #1 - “What Is Right For Me?” workbook – and also product #2 - “Supporting Choice” workbook. The second phase will test product #3 - the computer-based Decision Aid on Antipsychotic Medications.

Phase 1: In the first phase, consumers of mental health services will be asked to use the product #1, and mental health service providers will be asked to use product #2, as well as to assist consumers as requested with the use of product #1, and to meet with consumers who have used product #1. Consumers and providers who have used the products will be asked to complete Surveys #1 and #2 respectively.

Phase 2: In the second phase, consumers will be asked to use product #3, and providers will be asked to meet with consumers who have used product #3 as well as to review product #3 themselves. Consumers and providers who have used product #3 will be asked to complete Surveys #3 and #4 respectively.

Identification of respondents

The recruitment procedures at the three test sites will consist of the following:

• Orientation of consumers of mental health services and mental health service providers to the three products and the concept of shared decision-making;

• Promotion and distribution of product #1 and #2 by designated agency staff, including peers, through existing forums such as support groups, group and individual therapy sessions, peer support forums, and staff meetings;

• Promotion of product #3 through the same mechanisms and including information about how to obtain electronic access;

• Testing site staff will track consumers and providers who take a copy of product #1 or #2 or use product #3.

• Consumers and providers who used any of the products will be invited to complete group-administered surveys.

Proposed Sample

The three test sites are in different geographical locations and have a diverse range of programs serving consumers. They are large enough to support the sample numbers described here.

At each site, approximately 15 groups will be scheduled for consumers to complete the surveys. Each of the 15 groups will consist of 6 to 10 consumers for a total of approximately 120 consumer respondents per site per phase or a total of 720 consumer respondents.

Site 1: 15 groups x 6-10 consumers x 2 phases = 180 - 300 respondents (approximately)

Site 2: 15 groups x 6-10 consumers x 2 phases = 180 - 300 respondents (approximately)

Site 3: 15 groups x 6-10 consumers x 2 phases = 180 - 300 respondents (approximately)

Total for all 3 sites = approximately 720 consumer respondents

Similarly, at each site approximately 5 groups will be scheduled for providers where surveys will be administered to 4-6 individuals for a total of 25 per site per phase or a total of 150 provider respondents.

Site 1: 5 groups x 4-6 providers x 2 phases = 40 - 60 respondents (approximately)

Site 2: 5 groups x 4-6 providers x 2 phases = 40 - 60 respondents (approximately)

Site 3: 5 groups x 4-6 providers x 2 phases = 40 - 60 respondents (approximately)

Total for all 3 sites = approximately 150 provider respondents

Some overlap may occur; respondents could potentially complete both the workbook and the decision aid survey at different times. However, they would be counted twice in the numbers described above and in the response burden estimates below.

Data collection

Use of product #1 and product #2 will be promoted at each of the three sites over the course of a two – three month period to enable consumers and providers an opportunity to review and use the products.

At two time points, four – six weeks after the initial orientation and distribution of the products, and again at eight – twelve weeks, groups for survey administration with consumers and with providers will be scheduled on site. The testing site staff will contact consumers and providers who used products #1 and #2 and expressed interest in the survey and invite them to the groups to complete surveys. Only the testing sites will track individuals; the evaluator will not have this information. This process will be repeated for product #3 over a subsequent three month period.

Methods used to protect the identity of respondents

Names will not be collected or ever associated with surveys. Evaluator staff members involved in data collection will sign statements pledging not to disclose any information about participants. Survey data will be entered into electronic files using software with security features and with access limited to evaluator staff involved in data entry and analysis. All electronic data will be kept in a secure folder. All electronic files will be kept on a secure network drive only accessible with a user ID and password. The folder holding all documents will have an extra layer of security, allowing only certain individuals access as designated by the evaluator. All surveys will be stored in a locked file with access restricted to the evaluator. Once the records are in electronic form, they will be destroyed, unless there are parts that are not entered into an automated file. Physical and electronic copies of survey data will be destroyed on completion of the Satisfaction Survey.

Response rate

Response rates to face-to-face administration of surveys is high, generally 80% to 85% (Punch 2003). SAMHSA expects response rates to be on the high end of this range as individuals will attend groups with the expectation of completing surveys. Based on average group attendance (see description under “Proposed Sample”) this response rate will result in the following number of consumers of mental health services completing surveys: between 576 (80% of 720) and 612 (85% of 720) or approximately 600 respondents. Similarly, expected number of test site staff completing surveys is as follows: between 120 (80% of 150) and 128 (85% of 150) or approximately 124 respondents.

Analyses

Survey data will be entered and analyzed in SPSS. Descriptive quantitative analyses will present the numbers of respondents and frequencies and/or means and standard deviations of specific survey items. Statistical analyses (independent t tests) will be conducted to examine differences in responses among any distinct demographic categories (e.g., gender) and between sites.

D. Annual (2010) response burden estimate

Surveys	Number of Respondents (a)	Responses per Respondent	Hours per Respondent (b)	Annual Burden Hours	Hourly wage (c)	Total Hourly Cost
Consumer Surveys	600	1	.25	150	$7.25	$1088.00
Staff Surveys	124	1	.17	21	$25.00	$525.00
TOTAL	724			171		$1613.00

(a) There will be three survey sites. Consumer surveys are expected to be completed by 100 consumers per site for product #1 and 100 consumers per site for product #3. Provider surveys are expected to be completed by 20 staff per site for products #1 and #2 and 20 staff per site for product #3. See description above in Section C “Overview of data collection methodology: Proposed sample”.

(b) The average time to complete each survey for consumers is estimated to be 15 minutes or .25 hours. The average time to complete each shorter survey for providers is estimated to be 10 minutes or .17 hours.

(c) The minimum wage rate of $7.25 was used to calculate the hourly burden for consumer surveys. Hourly wage rate for providers was estimated as $25.00/hour based on an average annual salary for social service agency provider of $50,000 (as estimated for SAMHSA’s Treatment Episode Date Set (TEDS), OMB No. 0930-0106).

E. Methods used to develop and test the questions

The surveys were developed specifically to assess the satisfaction of consumers of mental health services and mental health service providers with three products designed to support shared decision-making in mental health: (1) the “What Is Right For Me?” Workbook, (2) the “Supporting Choice” Workbook, and (3) the computer-based Decision Aid on Antipsychotic Medications. The survey questions were designed to assess whether the goals which guided the development of the decision support tools were met. Standard approaches for the design of survey items were used. Related instruments were reviewed and contributed to survey and item development (Hibbard, et al., 2004; Green, et al., 2009; Salyers, et al., 2009).

F. Federal Project Officer

Stacey Lesko, Consumer Affairs Specialist

SAMHSA/CMHS

1 Choke Cherry, Rm. 6-1070, Rockville, MD 20857 (postal mail) or 20850 (FedEx/UPS/DHL)

240-276-1390

Stacey.Lesko@samhsa.hhs.gov

 

G. Project evaluator and data collection entity

John A. Hornik, Ph.D.

Director of Research

Advocates for Human Potential, Inc.

324 Broadway, 2 E-Comm Sq.

Albany, NY 12207

(518) 729-1244

[email protected]

References

Green, C.A., Perrin, N.A., Polen, M.R., Leo, M.C., Hibbard, J.H., and Tusler, M. (2009). Development of the patient activation measures for mental health. Administration and Policy in Mental Health. Published online: 29 August 2009.

Hibbard, J.H, Stockard, J., Mahoney, E.R., and Tusler, M. (2004). Development of the patient activation measure (PAM). Health Services Research 39(4), 1005-1026.

Punch, K.F. (2003). Survey Research: The Basics. London: Sage Publications Ltd.

Salyers, M.P., Matthias, M.S., Spann, C.L., Lydick, J.M., Rollins, A.L., and Frankel, R.M. (2009). The role of patient activation in psychiatric visits. Psychiatric Services 60(11), 1535-1539.

List of Attachments

1. Product #1 – “What Is Right For Me? How to Make Decisions in Everyday Life”

2. Product #2 – “Supporting Choice: Helping Others Make Important Decisions”

3. Survey #1 – What Is Right For Me Workbook User Satisfaction Survey for Mental Health Services Consumers

4. Survey # 2 – Decision Support Workbook User Satisfaction Survey for Mental Health Services Providers

5. Survey #3 – Decision Aid on Antipsychotic Medications User Satisfaction Survey for Mental Health Services Consumers

6. Survey #4 – Decision Aid on Antipsychotic Medications User Satisfaction Survey for Mental Health Services Providers

7