Supporting Statement

SAMHSA/CSAT Knowledge Application Program

Product and Customer Surveys

1. Product/Activity To Be Assessed

Synopsis. This is a request for approval to conduct two customer surveys (Product Feedback Survey [PFS] and Customer Information Survey [CIS]) to explore satisfaction with, and utilization of, information products developed by the Center for Substance Abuse Treatment’s (CSAT’s) Knowledge Application Program (KAP). PFS will assess content, format, accessibility, appearance, perceived usefulness, and dissemination methods of products, as well as customer job field, location, and demographics. CIS will assess customer product resource needs, preferred dissemination sources and methods, product formals, and demographics. The information obtained from both surveys will help enhance the accessibility of KAP products, expand the audience of potential users, and improve satisfaction with and use of KAP products.

General categories of products produced by KAP are described below.

1. Treatment Improvement Protocols (TIPs), currently being rebranded as Toolkits for Improving Practice: Developed through a consensus panel process, TIPs contain evidence-based and best practice guidelines. Each TIP is a comprehensive source document focusing on a specific topic of interest and concern to treatment providers and/or professionals working with clients/patients who may have substance use disorders, mental disorders, or both.

2. Other Source Documents: KAP also produces other source documents, including comprehensive treatment manuals for providers (e.g., the five-volume Cannabis Youth Treatment Series); technical assistance documents focusing on the treatment needs of specific populations (e.g., Clinical Drug Testing in Primary Care); curricula (e.g., curricula for conducting in-service training for staff working in therapeutic communities); and reports on promising practices (e.g., Pathways to Healing, a report on CSAT’s Rural, Remote, and Culturally Distinct Populations Program).

3. Collateral Products: Collateral products summarize information contained in the source document, are targeted to a specific audience (e.g., counselors), and may utilize a different medium than the source document. Designed to make information from the source document more readily and easily accessible, collateral products include quick guides for clinicians, program administrators, and specific professional groups; a series of quick guides on legal issues for substance use treatment providers; laminated assessment and screening instruments; brief modules for use in providing staff in-service training; brochures for clients and family members; and client workbooks. Common graphic elements link the collateral products with the source document, providing the basis for user recognition of each specific product family.

4. Other Products: KAP also produces other products, such as the Advisory and In Brief series. These other products all provide information about prevention and treatment services for substance use and mental disorders. Each product is developed for a specific targeted audience.

Assessing customer use of and satisfaction with KAP information products will enable CSAT to enhance the development, dissemination, and use of KAP products by members of the target audiences. The long-run objective is to enhance the likelihood that members of the target audiences will obtain CSAT’s knowledge application products, read them, and then implement the consensus- and evidence-based guidance these products promote.

1. Brief Description: Summary of Any Prior Customer Information Assessments of Similar CSAT Projects/Statement of Objectives of the Proposed Customer Information Effort

Prior KAP Customer Assessment Initiatives. Between 2004 and 2009, KAP conducted three customer information surveys for KAP Keys, the In Briefs, and the Advisories. The surveys sought to assess the target audiences’ opinion of the publications as a resource; learn about respondents’ access to, awareness of, attitudes toward, and use of the publications; identify how respondents typically obtain substance use treatment information; determine respondents’ previous awareness of CSAT products; identify the most frequently used CSAT publications; and gather suggestions on how best to ensure that the publications reach substance use treatment providers. Over 470 respondents completed one of the KAP surveys evaluating their satisfaction with KAP Keys, In Briefs, and Advisories. The majority of participants were randomly chosen from a list provided by SAMHSA’s Health Information Network (SHIN); the one exception was that KAP Keys respondents were conference attendees. The completed KAP customer information survey reports provided a summary of demographics, responses, and implications.

In 1997, CSAT funded the TIPs Evaluation Project, which consisted of a series of studies¹ that examined treatment providers’ and others’ awareness of, attitude toward, and use of TIPs. Data were also collected on how TIPs could be improved, and in one small study conducted under the project, the effectiveness of three alternative formats for information delivery (i.e., a TIP, a desk reference, and a four-page brochure) was assessed.

In 2015-2016, KAP conducted a customer information survey, assessing (1) the KAP target audience’s satisfaction with KAP products; (2) suggested changes to improve KAP products; (3) dissemination methods and sources utilized; and (4) effectiveness of dissemination techniques. The survey was distributed to a nonprobability, convenience sample of 500 members of KAP’s target audiences. Feedback from survey respondents and a continually increasing list of possible product formats, sources, and dissemination methods, made it clear that two questionnaires would be better implemented and received. This would allow for each survey to have a more directed focus – one on customer product satisfaction/feedback, and one on customer information needs/preferences. Thus, evolved the PFS and CIS.

CSAT’s ability to collect consumer preference data on products is limited to reviews of documents conducted by members of the target audiences prior to publication. These reviews focus primarily on content. CSAT has little opportunity to collect market data during or after the product development process, which curtails their ability to make changes in format, appearance, graphics, and other aspects of a document that might attract potential users’ attention at the point in time when modifications could be made most easily and cost effectively. The proposed data collection mechanisms will help CSAT address these problems.

Objectives of Proposed Customer Information Data Collection Efforts. Data collected will be used by CSAT’s KAP to enhance products—specifically their usability, formats, content, cultural responsiveness, accuracy, and accessibility—thereby increasing behavioral health services providers’ knowledge of and ability to institute evidence-based treatment approaches. Data collected will also be used by CSAT’s KAP to better understand who is utilizing KAP products, the source and method of customers’ exposure to products, and how products are being utilized, thus improving CSAT’s ability to expand the reach and focus of KAP product marketing. These objectives, in turn, will facilitate accomplishment of CSAT’s mandate to expand the availability of effective treatment and recovery services.

Specifically, CSAT seeks information about:

1. Customers’ satisfaction with the content, format, appearance, and accessibility of products.

2. Customers’ satisfaction with the usefulness of products.

3. The source and format of product notifications.

4. Accessibility programs or tools used to access content

5. Changes in the content, format, appearance, and usability of products that would better address customers’ needs and increase the likelihood that providers will adopt the knowledge and best practices contained in the products.

6. Purpose of customers’ product utilization.

7. How customers gather and assess new information in their field.

8. Preferred resource types and formats.

9. Professional context of product utilization (e.g. customers’ job fields, position, population served, geographic area)

CSAT will use this information to improve its products to better meet the needs of its audiences, and to more effectively disseminate its products to its audiences. These audiences include substance use treatment and mental health services providers; those working in prevention, peer support, criminal justice, family and child welfare, and primary care services; administrative and direct service or front-line staff who come into contact with individuals who have substance use, mental, and co-occurring disorders.

Analysis Plan. The proposed instruments are provided in Attachment 1 (PFS) and Attachment 2 (CIS). The main analyses of both surveys will examine respondents’ answers to the closed and open-ended questions. Each question will directly assess one of the nine objectives listed above.

A statistical software package, SPSS (v. 22), will be used to conduct analysis on all quantitative survey items. For each closed-ended item on the questionnaire, basic descriptive statistics will be calculated. Frequency distributions (raw numbers and percentages) will be presented for nominal and ordinal level variables. Measures of central tendency (e.g., mean, median, mode, standard deviation, and range) will be presented for interval and ratio level variables. As a general practice, means and standard deviations will be used for normally distributed variables. Medians will be used to show central tendency for skewed distributions. After conducting descriptive statistics, inferential statistics will be conducted when appropriate. Some analyses may include comparing different groups or trying to make predictions based on collected data. Analyses may include chi-square, analysis of variance (ANOVA), t-tests, and regressions when appropriate.

Finally, a content analysis of any open-ended items on the questionnaire will be conducted using ATLAS.ti, a qualitative software package. The software will be used by analysts to identify salient themes within the open-ended responses using a structured protocol designed to minimize potential investigator bias. Suggestions for improving the content, format, usability, and accessibility of CSAT’s KAP product offerings will be obtained through this analysis.

1. Overview of Methods To Be Used To Collect the Information

Data Collection Method. Two cross-sectional questionnaires will be used to obtain the product feedback and customer information data through online surveys, employing the SelectSurvey platform. Participants will include behavioral health and addiction treatment providers and program administrators, addiction-focused psychologists and psychiatrists, drug and alcohol recovery peer support specialists, as well as others who see patients with substance use disorders (primary care providers, criminal justice workers, employee assistance program specialists, family and child welfare workers, social workers, community health specialists, tribal leadership, etc.). Both surveys will begin with general information highlighting the source of the survey, purpose, privacy, and other informed consent content and issues relevant to this satisfaction survey. Consent is implied when a participant takes the survey.

PFS: When any of the KAP products are downloaded, potential participants will see a link to the survey website within each PDF file.

CIS: Potential participants will receive an e-mail that briefly explains the purpose of the survey and a link to the survey website.

Method for Sample Selection.

PFS: The survey will be included on the PDFs of up to 60 active KAP publications. Thus, the sampling universe will be the total of 500 participants who agree to take the product feedback survey upon downloading the PDF products. Potential respondents will include any user of a KAP product—substance use disorder prevention and treatment providers and mental health services providers, including clinicians, administrators, primary care providers, and front-line staff. By offering the survey to every user of these products, the feedback data will be as closely representative as possible of the characteristics of the surveyed population, to allow for results that can be generalized to all KAP product users.

CIS: A nonprobability, quota-based convenience sampling methodology will be used to select the sample for the CIS. The sampling universe will be the total of 500 participants who agree to take the satisfaction survey through an online request disseminated by the members of The KAP Target Audience Advisory Group (TAAG). TAAG is a group established by SAMHSA that consists of representatives from 16 substance abuse-related associations. Each of the 16 associations will send an email invitation to 75 randomly selected staff and constituent members that briefly explains the purpose of the survey and provides the survey link. The survey will stay active until 500 respondents have completed the survey. Potential respondents will be members of KAP’s target audiences—substance use disorder prevention and treatment providers and mental health services providers, including clinicians, administrators, primary care providers, and front-line staff. Although nonprobability samples preclude making statistical inferences or generalizing results to all CSAT customers, in this instance a convenience sample presents the best value to the government: the goal of the data collection effort is to create more successful products by eliciting customer input, not to measure, in a statistically representative way, satisfaction with CSAT products. Although nonprobability samples preclude making statistical inferences or generalizing results to all CSAT customers, in this instance a convenience sample presents the best value to the government: the goal of the data collection effort is to create more successful products by eliciting customer input, not to measure, in a statistically representative way, satisfaction with CSAT products.

Proposed Sample Size and Rationale.

PFS: The ongoing survey will stay open indefinitely, to assess the changing feedback of KAP product users, until 500 responses have been received. The PFS links will be added to all current KAP publications, and will be included with any new KAP products are they are published (until a total of up to 60 products are reached). This allows CSAT a steady stream of product feedback, both publication-unique and program-wide, throughout the duration of the product’s availability. This ensures that CSAT can be on top of the changing needs and preferences of their customers.

CIS: Five hundred respondents will be contacted to complete the CIS. This sample size was selected because it will provide CSAT with an appropriate balance of information quality and quantity to cost and time expended. The patterns of responses apparent in the opinions of participants will greatly enhance CSAT’s ability to improve KAP products without jeopardizing its ability to deliver these products in a reasonable amount of time and for a reasonable cost.

Because a nonprobability sampling methodology is being proposed for both surveys, and the sampling frame often will be unenumerated, it is not possible to conduct a power analysis to determine the appropriate sample sizes.

Planned Frequency of Information Collection.

PFS: Each respondent will be asked to complete a PFS only once per downloaded product.

CIS: Each respondent will be asked to complete a CIS only once.

Time Period Over Which Information Will Be Collected.

PFS: Data collection will be ongoing, for an indeterminate period of time. Survey will be re-assessed annually for necessary changes or technical updates.

CIS: Data will be collected annually

Expected Response Rate and Plan for Follow-Up, If Any, of Non-Respondents. Response rates typically are not calculated for convenience samples, particularly those employing quotas, in part because sampling frames upon which to calculate a response rate often are unenumerated, and in part because interpreting the response rate would be difficult given that not every person had an equal chance of being sampled. There is no plan to follow up with individual non-respondents. However, if the flow of survey data is slower than anticipated and it seems unlikely to reach the goal of 500 responses, the 16 TAAG groups may be asked to send out survey request e-mails to another selection of their members and/or staff.

Expected Ability To Assess Non-KAP and Non-Response User Bias.

PFS: As all survey respondents will be individuals who have downloaded a KAP publication, all respondents should have familiarity with KAP product offerings. Based on the data of the number of product downloads and the number of surveys completed, CSAT will assess the number of those who choose not to complete the survey on SelectSurvey.

CIS: It is reasonable to expect that respondents to the CIS may not be familiar with KAP product offerings, which does not jeopardize the study or the utility of the data collected. In fact, this bias may work to CSAT’s benefit in that those who agree to participate in the survey will provide information on how they obtain information related to their field as well as their preferences for how they would like to receive professional and educational materials. Based on how many individuals received the e-mail invitation, analysts will assess the number of those who choose not to complete the survey on SelectSurvey.

Methods Used To Maintain Customer Privacy. In order to ensure customer privacy, respondents will not be asked to provide any personal information such as name or organizational affiliation. Online results of the surveys, without respondent identifying information, will be stored by the Contractor for a period of 5 years.

3. Annual Response Burden Estimate

PFS: Burden for respondents includes the time and cost estimates for completing the product feedback survey. The total response burden estimate is 58 hours for this survey. Each of the 500 respondents will be asked to complete one survey estimated to take 7 minutes (.116 hours) to complete, including the time for reviewing instructions. Time estimates for the survey were obtained by taking the average time it took pilot study participants to complete the survey.

The total burden cost estimate for all respondents of this survey is $2,288.68. The total hourly wage burden estimate ($39.46 per hour) was calculated by averaging the mean hourly wage estimates for 13 occupational categories listed in the most recent National Occupational Employment and Wage Estimates provided by the Bureau of Labor Statistics (BLS).² These 12 BLS occupational categories typify the audiences for whom CSAT products are developed.

To obtain the total burden cost estimate for the customer information survey, the total hourly wage burden estimate was multiplied by the total response burden hour estimate for the project (i.e., $39.46 x 58 hours = $2,288.68). The total burden cost estimate will involve no startup or operational/maintenance cost to respondents.

CIS: Burden for respondents includes the time and cost estimates for completing the customer information survey. The total response burden estimate is 50 hours for this survey. Each of the 500 respondents will be asked to complete one survey estimated to take 6 minutes (.100 hours) to complete, including the time for reviewing instructions and reviewing sample products. Time estimates for the survey were obtained by taking the average time it took pilot study participants to complete the survey.

The total burden cost estimate for all respondents of this survey is $2,170.00. The total hourly wage burden estimate ($39.46 per hour) was calculated by averaging the mean hourly wage estimates for 13 occupational categories listed in the most recent National Occupational Employment and Wage Estimates provided by the Bureau of Labor Statistics (BLS).³ These 13 BLS occupational categories typify the audiences for whom CSAT products are developed.

To obtain the total burden cost estimate for the customer information survey, the total hourly wage burden estimate was multiplied by the total response burden hour estimate for the project (i.e., $39.46 x 50 hours = $1,973.00). The total burden cost estimate will involve no startup or operational/maintenance cost to respondents.

Instrument/ Activity	Number of Respondents	Responses per Respondent	Total Number of Responses	Hours per Response	Total Burden Hours	Wage Cost	Total Cost
SAMHSA/CSAT Knowledge Application Program Customer Information Survey	500	1	500	.100	50	$39.46	$1,973.00
SAMHSA/CSAT Knowledge Application Program Product Feedback Survey	500	1	500	.116	58	$39.46	$2,288.68
Total	1,000		1,000		108		4,261.68
Note: Estimates were obtained from the Bureau of Labor Statistics, 2016 National Occupational Employment and Wage Estimates.

3. Methods Used To Develop and Test the Questions

Questionnaire items were designed to meet CSAT’s goal of identifying changes to products that could increase consumer satisfaction with these products. The nine areas of interest, presented in Section B above, guided the development of both questionnaires. Previous KAP surveys, previous customer satisfaction surveys by other government agencies, and Office of Management and Budget (OMB) recommendations for customer satisfaction surveys of government programs were reviewed and used to develop the surveys.

The questionnaires have been kept as short as possible to make recipients more receptive to completing them, while still ensuring that CSAT obtains useful and informative data. The questions and instructions were carefully designed and ordered to make administration and completion of the survey easier. Please note that to reduce burden on respondents, when completing the PFS in the online system, the participant first selects a broad category (shown as bolded and underlined in the paper version). Then, a short list of products is displayed (shown in the paper version as the list of options underneath that category name), from which the user can select the product they recently accessed.

The questionnaire was pilot tested by four contractor staff. Based on the results of pilot testing, minor revisions were made to some response options in order to facilitate respondent understanding. None expressed any difficulty in answering the questions, nor did any ask for clarification as to the meaning of any of the questions. On average, the respondents answered all questions in approximately 6 minutes.

3. Name, Title, and Organizational Affiliation of Project Statistician

The representative of the Contractor responsible for conducting the planned data collection and analysis is:

Larry Robertson

Vice President

JBS International, Inc.

5515 Security Lane, Suite 800

North Bethesda, MD 20852

240-645-4520

Candi Byrne

SAMHSA/CSAT

5600 Fishers Lane

13E05-B

Rockville, MD 20857

301-276-1572

3. Attachments

1. Customer Information Survey

2. Product Feedback Survey

1These studies were conducted under OMB approval numbers 0930-0189, 0930-0212, 0930-0224, and 0930-0198.

2The most recent estimates were obtained from the Bureau of Labor Statistics, 2016 National Occupational Employment and Wage Estimates.

3The most recent estimates were obtained from the Bureau of Labor Statistics, 2016 National Occupational Employment and Wage Estimates.

5