Supporting Statement

BRSS TACS TA Satisfaction Survey

A. Product/Activity to Be Assessed

The Substance Abuse and Mental Health Services Administration’s (SAMHSA’s) Bringing Recovery Supports to Scale Technical Assistance Center Strategy’s (BRSS TACS) overarching goal is to expand the integration of recovery support and services through partnerships with people in recovery from mental illness and substance use disorders and their family members to guide the behavioral health system and promote individual, program, and system-level approaches that foster health and resilience. Training and technical assistance (TTA) shall serve as a primary source to provide information and the key audiences include: states and territories, tribes, counties, other health and related systems; providers, SAMHSA grantees, Consumer-Operated Services providers, certified peer specialists, and other peer providers; and other stakeholders. SAMHSA is requesting approval from the Office of Management and Budget (OMB) to implement a new data collection procedure with this satisfaction survey for participants of TTA events supported by the BRSS TACS contract. BRSS TACS considers it important to assess participants’ perception of the value of services provided in these events.

The Center for Mental Health Services and the Center for Substance Abuse Treatment (CMHS/CSAT) propose the use of 1 instrument for program monitoring of BRSS TACS TTA events as well as for ongoing quality improvement. This is the Training and Technical Assistance Post-Event Form.

Training events can take the form of webinars, workshops, conferences, and continuing education courses. While the majority of training events are done using technology such as online and video courses, some will be conducted in person. Training participants come from diverse populations, ranging from individual grantee data entry staff to SAMHSA Government Project Officers (GPOs) and program leadership. Technical assistance events are jointly planned consultations generally involving a series of contacts between BRSS TACS staff and an outside organization/institution during which the BRSS TACS staff provide expertise and gives direction toward resolving a problem or improving conditions. Technical assistance events will be conducted via email, teleconference, videoconference, or in person. Other meeting events are BRSS TACS-sponsored or co-sponsored events in which a group of people representing one or more agencies other than the BRSS TACS contract work, cooperatively on a project, problem, and/or a policy.

B. Brief Statement of Objectives

BRSS TACS TTA will use the satisfaction survey responses collected to guide decision making about the content and effectiveness of TTA events. The information gathered would help BRSS TACS identify areas needing improvement and implement changes that are practical and meet participants’ needs. BRSS TACS will use the knowledge and insight gained from the responses to plan and, if necessary, to redirect resources and efforts to improve or maintain a high quality of service to users of discretionary program performance data.

All of the information collected from participants is critical for assessing the effectiveness of the BRSS TACS TTA events. Without this information, CMHS/CSAT will be unable to:

• determine the satisfaction with BRSS TACS TTA events by participants; and

• determine the effectiveness of various training formats to transfer knowledge and skills to participants.

If BRSS TACS does not collect this information, vital feedback regarding participants’ perception of the value of the TTA will be unavailable. This missed opportunity will hinder BRSS TAC’s ability to develop, implement, and refine TTA materials and content in a manner best tailored for discretionary programs’ needs.

Event Definitions

The definitions for the two types of events from which data will be collected are as follows:

Training Event—A training event is defined as a BRSS TACS sponsored or co-sponsored event that focuses on the enhancement of knowledge and/or skills.

Technical Assistance—Technical assistance is defined as a jointly planned consultation generally involving a series of contacts between the BRSS TACS staff and an outside organization/institution. During the consultation, the BRSS TACS staff provides expertise and gives direction toward resolving a problem or improving conditions. This may be a time-limited consultation or an ongoing series of consultations. The BRSS TACS staff report technical assistance for more intensive engagements at the end of the series of contacts, with all TA contact summary data provided monthly or yearly.

C. Overview of Methods to Collect Information

1. Data Collection Method

Data collected on the forms will be entered into an online system maintained by a CMHS/CSAT contractor. Data entered into this online system are periodically reported to CMHS/CSAT Project Officers for administration purposes. As described previously, CMHS/CSAT requests the use of one (1) form to monitor the work of the BRSS TACS subcontractors.

Event description data will continue to be reported by BRSS TACS staff on both types of events using the first two questions on the Post-Event Form. It allows the BRSS TACS contractors and CMHS/CSAT to track the number and types of events held (see Attachment 1).

Post-event data will be collected on participants of both types of events according to CMHS/CSAT requirements using one proposed form.

Post-Event Form for Training and Technical Assistance: The Training and Technical Assistance Post-Event Form, which is administered immediately following the event, currently asks approximately 11 questions of each individual that participated in the training (Attachment A). The instrument asks the participants to report workplace role, program, and satisfaction with the quality of the training and training materials.

Table 1: Data Collection Instruments

Form	Timeline	Type of Information
Participants
Post-Event Form for Training and Technical Assistance	Completion of each training or technical assistance event	The form asks participants to report event name and type, role, program, satisfaction with the quality of the training and training materials.

Use of Information Technology

Fewer than 20 percent of the BRSS TACS TTA performance-monitoring instruments are administered in person to participants at in-person BRSS TACS TTA events and these participants will complete the forms by paper and pencil. Approximately 80 percent of TTA events are online, and thus, those instruments are administered online. Data collection for the Post-Event instrument is incorporated into the event procedure.

All data collected will be managed in an electronic database. The BRSS TACS TTA contractors are responsible for data collection and entry for their events. Once data are entered into the system, they are available to CMHS/CSAT for review. The BRSS TACS TTA contractors can also download these data for their use.

2. Method for Identifying Respondents

BRSS TACS will respond to TA requests from a range of behavioral health organizations throughout the United States, territories, and tribal authorities through nine different tasks with additional TA task added in Option Year (OY)1. Each person participating in a TA event will be asked to complete a TA satisfaction survey.

BRSS TACS sends the TA satisfaction survey to all participants, including grantee leadership and SAMHSA Government Project Officers (GPOs), to provide feedback.

BRSS TACS will send the survey via email to all TA participants. At the end of a virtual training, the TA satisfaction survey will be posted in the online meeting room and participants will be directed to complete the survey immediately following the event. We will also send these participants a follow-up email with the TA satisfaction survey to complete. Non-respondents will be sent two reminder emails plus a personal email from a BRSS TACS staff person requesting that they respond.

Participation will be voluntary, and not completing the satisfaction survey will have no effect on receiving Continuing Education Unit (CEU) credits or a course completion certification. The surveys are anonymous with no identifying information. In addition, a BRSS TACS partner agency will analyze and summarize the TA satisfaction surveys, further protecting confidentiality.

Respondents will not have to download and save any data onto their own computer.

3. Proposed Sample Size and Rationale

BRSS TACS estimates that approximately 500 individuals per year will have occasion to complete a survey at the conclusion of a TA engagement or event.

4. Planned Frequency of Information Collection

Data collection will occur as individuals participate in BRSS TACS TTA events. Because this assessment is used to monitor and improve upon the quality of BRSS TACS TTA services, ongoing examination is critical. The BRSS TACS contractor maintains an electronic database in which the data will continue to be entered allowing reports to be run on the data in a quick and timely manner. At least once a year the BRSS TACS contractor will send the BRSS TACS COR aggregated, contract-wide data on the following categories:

• Total events

• Total events in each of the two event categories

• Total participants

• Total participants in each of the two event categories

• Percentages of participants of each role (grantee, SAMHSA staff, etc.) combined for both event types

• Percentages of participants who are overall satisfied with BRSS TACS TTA events taken as a whole

• Percentages of participants who are overall satisfied with each of the two types of events

• Percentages of participants who expect to use information from all BRSS TACS TTA events

• Percentages of participants who expect to use information from each of the two types of events

• Percentages of participants who learned something valuable from all BRSS TACS TTA events combined

• Percentages of participants who learned something valuable from each of the two types of events

The BRSS TACS contractors will present the data in a brief, easy-to-read format for dissemination to key stakeholders.

5. Methods for Identifying Duplications

The data to be collected are unique and are not otherwise available.

6. Time Period Over Which Information Will Be Collected

Data collection will occur from clearance of the instrument through the end of the contract’s performance period, if optioned, to September 2020.

7. Expected Response Rate and Plan for Follow-up for Non-respondents

A response rate of 40 to 60 percent is expected. BRSS TACS will send three rounds of prompts to non-respondents; the original round will immediately follow webinars, modules, and virtual trainings (including telephonic TA). A reminder will be sent the following day, and a second reminder a week later. Follow up reminders for in-person events will occur three and five days following the events. Cross-referencing respondents against the universe of individuals to whom BRSS TACS emailed the survey link will identify these non-respondents. There will be no follow up for paper-based surveys.

8. Expected Ability to Assess Non-Response Bias Using Existing Information

A response rate of 40 to 60 percent is expected. BRSSS TACS does not expect nonresponse bias, but knowledge of some characteristics of non-respondents based on pre-sample information will allow us to determine whether non-respondents are distinct with regard to program experience, program type, or Center.

9. Methods Used to Maintain Customer Privacy

No forms collect personally identifying information. Information that would allow individuals to be identified through combinations of characteristics (e.g. where a program has only one GPO) will be suppressed in reporting. Additionally, no forms collect information that is sensitive to individuals.

D. Annual Response Burden Estimate

The total annualized burden to an estimated 500 respondents for BRSS TACS TTA satisfaction data collection is estimated to be 100 hours. The BRSS TACS TTA contractors base burden estimates on cognitive/pilot testing of instruments. The annualized hourly costs to respondents are estimated to be $2,064.00. Hourly wage information is based on estimated median hourly wages of $20.64 an hour for substance abuse and behavioral disorder counselors as reported in the Occupational Employment Statistics available from the Bureau of Labor Statistics, U.S. Department of Labor (http://www.bls.gov/oes/current/oes211011.htm). There are no direct costs to respondents for participation aside from their time. Burden estimates are detailed in Table 2.

Table 2: Annualized Burden Estimates

Type of Respondent	Number of Respondents	Responses per Respondent	Total Responses	Hours per Response	Total Annual Burden Hours	Hourly Wage Cost	Total Hour Cost
Post-Event Form	500	1	500	.2	100	$20.64	$2,064.00

Estimates of Annualized Cost Burden to Respondents

There are neither capital or startup costs nor are there any operation and maintenance costs.

Estimates of Annualized Cost to the Government

The annual estimated cost to the government for the BRSS TACS program is $3 million. Approximately $78,234 per year represents SAMHSA costs to manage/administrate the program for 90 percent of one employee (GS-13).

E. Methods Used to Develop and Test the Questions

BRSS TACS developed the TA satisfaction survey through collaboration with TA providers, TA participants, SAMHSA personnel, and research experts. SAMHSA Contracting Officer Representatives (CORs) reviewed each round of survey development.

BRSS TACS piloted the survey with a BRSS TACS staff and a sample of 5TA participants.

F. Consultants within SAMHSA/PN and Outside the Agency

The consultant within SAMHSA for the survey is Sharon Amatetti, who also serves as the Alternate Contracting Reporting Officer (ACOR) for the BRSS TACS contract. Ms. Amatetti’s contact information is

Sharon Amatetti, MPH

BRSS TACS

Alternate Contracting Reporting Officer and CSAT Liaison

SAMHSA

5600 Fishers Lane

Rockville, MD 20857

(240) 276-1600

[email protected]

The BRSS TACS team has access to the statistical expertise required for the data analysis. BRSS TACS is a project housed at the Center for Social Innovation (C4). Dr. Bethany Marcogliese, a C4 associate, will provide the statistical tasks for this project.

Bethany Marcogliese, Ph.D.

Center for Social Innovation

200 Reservoir Street, Suite 202

Needham, MA 02494

781-247-1784

[email protected]

G. List of Attachments

Attachment A: Satisfaction Survey

Attachment B: Email to respondents

Attachment C: Screenshots of Survey

7