Justification

SAMHSA/CSAT Knowledge Application Program

Product Customer Satisfaction Survey

1. Product/Activity To Be Assessed

Synopsis. This is a request for approval to conduct a customer survey to explore satisfaction with information products developed by the Center for Substance Abuse Treatment’s (CSAT’s) Knowledge Application Program (KAP). Content, format, appearance, perceived usefulness, and dissemination methods of products will be assessed. The information obtained will help enhance the accessibility of KAP products, expand the audience of potential users, and improve satisfaction with and use of KAP products.

Background. CSAT was created by congressional mandate to improve the availability of treatment for substance use disorders. In keeping with this mandate, CSAT has focused its contracts, grants, and cooperative agreements on developing knowledge about effective best practices. In 1999, CSAT created KAP, assigning it responsibility for capturing the knowledge developed through CSAT’s broad array of funded initiatives, packaging and disseminating this knowledge, and supporting activities that encourage the adoption of best practice guidelines by substance use treatment providers and professionals working in related fields. In recent years, CSAT has expanded KAP’s scope of content to include mental health and prevention topics, and has broadened KAP’s audience to include mental health services providers.

General categories of products produced by KAP are described below.

1. Treatment Improvement Protocols (TIPs): Developed through a consensus panel process, TIPs contain evidence-based and best practice guidelines. Each TIP is a comprehensive source document focusing on a specific topic of interest and concern to treatment providers and/or professionals working with clients/patients who may have substance use disorders, mental disorders, or both.

2. Other Source Documents: KAP also produces other source documents, including comprehensive treatment manuals for providers (e.g., the five-volume Cannabis Youth Treatment Series); technical assistance documents focusing on the treatment needs of specific populations (e.g., Clinical Drug Testing in Primary Care); curricula (e.g., curricula for conducting in-service training for staff working in therapeutic communities); and reports on promising practices (e.g., Pathways to Healing, a report on CSAT’s Rural, Remote, and Culturally Distinct Populations Program).

3. Collateral Products: Collateral products summarize information contained in the source document, are targeted to a specific audience (e.g., counselors), and may utilize a different medium than the source document. Designed to make information from the source document more readily and easily accessible, collateral products include quick guides for clinicians, program administrators, and specific professional groups; a series of quick guides on legal issues for substance use treatment providers; laminated assessment and screening instruments; brief modules for use in providing staff in-service training; brochures for clients and family members; and client workbooks. Common graphic elements link the collateral products with the source document, providing the basis for user recognition of each specific product family.

4. Other Products: KAP also produces other products, such as the Advisory and In Brief series. These other products all provide information about prevention and treatment services for substance use and mental disorders. Each product is developed for a specific targeted audience.

Assessing customer use of and satisfaction with KAP information products will enable CSAT to enhance the development, dissemination, and use of KAP products by members of the target audiences. The long-run objective is to enhance the likelihood that members of the target audiences will obtain CSAT’s knowledge application products, read them, and then implement the consensus- and evidence-based guidance these products promote.

1. Brief Description: Summary of Any Prior Customer Satisfaction Assessments of Similar CSAT Projects/Statement of Objectives of the Proposed Customer Satisfaction Effort

Prior KAP Customer Satisfaction Assessment Initiatives. Between 2004 and 2009, KAP conducted three customer satisfaction surveys for KAP Keys, the In Briefs, and the Advisories. The surveys sought to assess the target audiences’ opinion of the publications as a resource; learn about respondents’ access to, awareness of, attitudes toward, and use of the publications; identify how respondents typically obtain substance use treatment information; determine respondents’ previous awareness of CSAT products; identify the most frequently used CSAT publications; and gather suggestions on how best to ensure that the publications reach substance use treatment providers. Over 470 respondents completed one of the KAP surveys evaluating their satisfaction with KAP Keys, In Briefs, and Advisories.

The majority of participants were randomly chosen from a list provided by SAMHSA’s Health Information Network (SHIN)”; the one exception was that KAP Keys respondents were conference attendees. The completed KAP customer satisfaction survey reports provided a summary of demographics, responses, and implications. No other studies or customer satisfaction assessments of any products developed under KAP or other CSAT-funded projects have been conducted since 2009.

In 1997, CSAT funded the TIPs Evaluation Project, which consisted of a series of studies¹ that examined treatment providers’ and others’ awareness of, attitude toward, and use of TIPs. Data were also collected on how TIPs could be improved, and in one small study conducted under the project, the effectiveness of three alternative formats for information delivery (i.e., a TIP, a desk reference, and a four-page brochure) was assessed.

Our ability to collect market research data on products is limited to reviews of documents conducted by members of the target audiences prior to publication. These reviews focus primarily on content. We have little opportunity to collect market research data during or after the product development process, which curtails our ability to make changes in format, appearance, graphics, and other aspects of a document that might attract potential users’ attention at the point in time when modifications could be made most easily and cost effectively. The proposed data collection mechanisms will help us address these problems.

Objectives of Proposed Customer Satisfaction Data Collection Efforts. Data collected will be used by CSAT’s KAP to enhance products—specifically their usability, formats, and accessibility—thereby increasing behavioral health services providers’ knowledge of and ability to institute evidence-based treatment approaches. This, in turn, will facilitate accomplishment of CSAT’s mandate to expand the availability of effective treatment and recovery services.

The goals of the data collection efforts described here are to gain insights into how to improve customers’ satisfaction with and use of KAP products. This information will be used to improve CSAT’s product offerings and dissemination efforts.

Specifically, CSAT seeks information about:

1. Customers’ satisfaction with the content of products.

2. Customers’ satisfaction with the format and appearance of products.

3. Perceived usefulness of the information provided in products.

4. Changes in the content, format, appearance, and usability of products that would better address customers’ needs and increase the likelihood that providers will adopt the knowledge and best practices contained in the products.

5. Effectiveness of dissemination methods to the target audiences.

6. Modifications to the dissemination process for a specific product to improve the effectiveness of dissemination methods.

CSAT will use this information to improve its products to better meet the needs of its audiences. These audiences include substance use treatment and mental health services providers; those working in prevention, peer support, criminal justice, family and child welfare, and primary care services; administrative and direct service or front-line staff who come into contact with individuals who have substance use, mental, and co-occurring disorders. In addition, CSAT will gain knowledge applicable to enhancing the development of products and identifying the most effective strategies for their dissemination.

Analysis Plan. The proposed instrument is provided in Attachment 1. The main analyses of the survey will examine respondents’ answers to the closed and open-ended questions. Each question will directly assess one of the six objectives listed above.

For each closed-ended item on the questionnaire, basic descriptive statistics will be calculated. Frequencies (raw numbers and percentages) will be presented for nominal and ordinal level variables. Measures of central tendency (e.g., mean, median, mode, standard deviation, and range) will be presented for interval and ratio level variables. As a general practice, means and standard deviations will be used for normally distributed variables. Medians will be used to show central tendency for skewed distributions.

Finally, a content analysis of any open-ended items on the questionnaire will be conducted. Suggestions for improving the content, format, usability, and dissemination method of CSAT’s KAP product offerings will be obtained through this analysis.

1. Overview of Methods To Be Used To Collect the Information

Data Collection Method. A cross-sectional questionnaire will be used to obtain the customer satisfaction data through an online survey, employing the SurveyMonkey platform. Participants will include behavioral health services providers (mental health services and substance use treatment providers) as well as those working in prevention, peer support, criminal justice, family and child welfare, and faith-based and primary care services. Potential participants will receive an email that contains a brief introduction and a link to SurveyMonkey. The survey will begin with general information highlighting the source of the survey, purpose, privacy, and other informed consent content and issues relevant to this satisfaction survey. Consent is implied when a participant takes the survey.

Method for Sample Selection. A nonprobability, quota-based convenience sampling methodology will be used to select the sample for the survey. The sampling universe will be the total of 500 participants who have agreed to take the satisfaction survey through an online request. Potential respondents will include members of KAP’s target audiences—substance use disorder prevention and treatment providers and mental health services providers, including clinicians, administrators, primary care providers, and front-line staff. Although nonprobability samples preclude making statistical inferences or generalizing results to all CSAT customers, in this instance a convenience sample presents the best value to the government: the goal of the data collection effort is to create more successful products by eliciting customer input, not to measure, in a statistically representative way, satisfaction with CSAT products.

Proposed Sample Size and Rationale. Five hundred respondents will be contacted to complete the KAP survey. This sample size was selected because it will provide CSAT with an appropriate balance of information quality and quantity to cost and time expended. The patterns of responses apparent in the opinions of participants will greatly enhance CSAT’s ability to improve KAP products without jeopardizing its ability to deliver these products in a reasonable amount of time and for a reasonable cost.

Because a nonprobability sampling methodology is being proposed, and the sampling frame often will be unenumerated, it is not possible to conduct a power analysis to determine the appropriate sample size.

Planned Frequency of Information Collection. Each respondent will be asked to complete a satisfaction survey only once.

Time Period Over Which Information Will Be Collected. Data will be collected annually.

Expected Response Rate and Plan for Follow-Up, If Any, of Non-Respondents. Response rates typically are not calculated for convenience samples, particularly those employing quotas, in part because sampling frames upon which to calculate a response rate often are unenumerated, and in part because interpreting the response rate would be difficult given that not every person had an equal chance of being sampled. There is no plan to follow up with non-respondents.

Expected Ability To Assess Non-KAP and Non-Response User Bias. It is reasonable to expect that respondents may not be familiar with KAP product offerings, which does not jeopardize the study or the utility of the data collected. In fact, this bias may work to CSAT’s benefit in that those who agree to participate in the survey will provide information on how they obtain information related to their field as well as their preferences for how they would like to receive professional and educational materials. Based on observation, we will record the number of those who choose not to complete the survey on SurveyMonkey.

Methods Used To Maintain Customer Confidentiality. In order to ensure customer confidentiality, respondents will not be asked to provide any personal information such as name or organizational affiliation. Online results of the surveys, without respondent identifying information, will be stored by the Contractor for a period of 5 years.

3. Annual Response Burden Estimate

Burden for respondents includes the time and cost estimates for completing the customer satisfaction survey. The total response burden estimate is 100 hours for this survey. Each of the 500 respondents will be asked to complete one survey estimated to take 12 minutes to complete, including the time for reviewing instructions and reviewing sample products. Time estimates for the survey were obtained by taking the average time it took pilot study participants to complete the survey.

The total burden cost estimate for all respondents of this survey is $4,234. The total hourly wage burden estimate ($42.34 per hour) was calculated by averaging the mean hourly wage estimates for 12 occupational categories listed in the most recent National Occupational Employment and Wage Estimates provided by the Bureau of Labor Statistics (BLS).² These 12 BLS occupational categories typify the audiences for whom our products are developed.

To obtain the total burden cost estimate for the customer satisfaction survey, the total hourly wage burden estimate was multiplied by the total response burden hour estimate for the project (i.e., $42.34 x 100 hours = $4,234). The total burden cost estimate will involve no startup or operational/maintenance cost to respondents.

Instrument/ Activity	Number of Respondents	Responses per Respondent	Total Number of Responses	Hours per Response	Total Burden Hours	Wage Cost	Total Cost
SAMHSA/CSAT Knowledge Application Program Product Customer Satisfaction Survey	500	1	500	.20	100	$42.34	$4,234.00
Note: Estimates were obtained from the Bureau of Labor Statistics, 2014 National Occupational Employment and Wage Estimates.

3. Methods Used To Develop and Test the Questions

Questionnaire items were designed to meet CSAT’s goal of identifying changes to products that could increase consumer satisfaction with these products. The six areas of interest, presented in Section B above, guided the overall questionnaire development. Previous KAP surveys, previous customer satisfaction surveys by other government agencies, and Office of Management and Budget (OMB) recommendations for customer satisfaction surveys of government programs were reviewed and used to develop the survey. Questions were taken directly from the KAP product satisfaction surveys, and modified in some areas to change an open-ended question to a close-ended question format.

The questionnaire has been kept as short as possible to make recipients more receptive to completing it, while still ensuring that CSAT obtains useful and informative data. The questions and instructions were carefully designed and ordered to make administration and completion of the survey easier.

The questionnaire was pilot tested by a KAP staff person who was also a member of CSAT’s target audience. None expressed any difficulty in answering the questions, nor did any ask for clarification as to the meaning of any of the questions. On average, the respondents answered all questions in approximately 12 minutes.

3. Name, Title, and Organizational Affiliation of Project Statistician

The representative of the Contractor responsible for conducting the planned data collection and analysis is:

Susan Hayashi, Ph.D.

Vice President

JBS International, Inc.

5515 Security Lane, Suite 800

North Bethesda, MD 20852

301-495-1080

The SAMHSA Project Officer is:

Suzanne D. Wise

Public Health Advisor

Center for Substance Abuse Treatment

Substance Abuse and Mental Health Services Administration

1 Choke Cherry Road, Room 5-1089

Rockville, MD 20857

240-276-1568

3. Attachment

Questions About KAP Products

1These studies were conducted under OMB approval numbers 0930-0189, 0930-0212, 0930-0224, and 0930-0198.

2The most recent estimates were obtained from the Bureau of Labor Statistics, 2014 National Occupational Employment and Wage Estimates.

3