Supporting Statement

SAMHSA Customer Satisfaction Survey

Center for Behavioral Health Statistics and Quality

Grantee Data Technical Assistance (GDTA)

Training and Technical Assistance (TA) Event Satisfaction

A. Product/Activity to Be Assessed

The Substance Abuse and Mental Health Services Administration’s (SAMHSA’s) Grantee Data Technical Assistance (GDTA) contract provides technical assistance (TA) for SAMHSA discretionary services grantees and project officers on data collection, management, and utilization. Designed to improve the accuracy, completeness, and timeliness of grantee data, GDTA promotes the use of data to improve practice, program design, and policy development. SAMHSA is requesting approval from the Office of Management and Budget (OMB) to implement a satisfaction survey for participants in TA and training events supported by the GDTA contract. GDTA considers it important to assess participants’ perception of the value of services provided in these events.

B. Brief Statement of Objectives

GDTA will use the satisfaction survey responses collected to guide decision making about the content and effectiveness of TA events. The information gathered will help GDTA identify areas needing improvement and implement changes that are practical and meet participants’ needs. GDTA will use the knowledge and insight gained from the responses to plan and, if necessary, to redirect resources and efforts to improve or maintain a high quality of service to users of discretionary program performance data.

If GDTA does not collect this information, vital feedback regarding participants’ perception of the value of the TA will be unavailable. This missed opportunity will hinder GDTA’s ability to develop, implement, and refine TA materials and content in a manner best tailored for discretionary programs’ needs.

C. Overview of Methods to Collect Information

1. Data Collection Method

GDTA will collect data at the completion of each TA engagement or event using either a paper or a web-based survey. Participants will have the option to decline responding to the survey. This survey will be used for longer-term TA events, not spot TA, such as that provided through Quick Answers.

Survey will include eleven (11) questions. The first question asks respondents to identify the TA activity for which they are providing information; wherever feasible this will be pre-populated. The second question, which will also be pre-populated wherever possible, asks for the type of event to be identified (examples include online training, site visit or other on-site TA, long-term telephone/email consultation, in-person conference presentation or workshop, annual or semi-annual grantee meeting presentation or workshop, or other event; in the case of “other,” a type can be specified by the respondent or pre-populated). The third question consists of a set of seven (7) scoring elements for aspects of the technical assistance event, with Likert-type scales for agreement or disagreement. The next section of the survey is applicable to products distributed as part of the TA activity, with questions asking the respondent to identify products and materials distributed with the activity, rating satisfaction with each product (using a Likert-type scale) and likelihood of distributing the products or materials within their organization. The next section of the survey asks about satisfaction with quality of instruction and overall satisfaction, using a Likert-type scale, and solicits information on the respondent’s preferences for topical areas of possible future trainings. The next section of the survey asks for a description of the respondents’ relationships to SAMHSA (grantees or staff, and their Center), and the grant(s) or grant program(s) with which they are associated.

Analyses will primarily consist of statistics to better understand the effectiveness and perceived value of the TA provided.

2. Method for Identifying Respondents

GDTA asks all participants in TA events, including grantee leadership and SAMHSA Government Project Officers (GPOs), to provide feedback by completing a customer satisfaction survey.

Participation will be voluntary, and not completing the satisfaction survey will have no effect on Continuing Education Unit (CEU) credits, or course completion certification. Given the voluntary nature of the data collection efforts, GDTA will make efforts to achieve the highest response rate possible.

GDTA will ensure the protection of respondents’ identifying information, and identifying information will not be included in the analysis or reporting of responses.

There will be no requirements for respondents to maintain or save any data onto their local computers.

3. Proposed Sample Size and Rationale

Based on the contractor’s experience with similar projects, GDTA estimates that approximately 500 individuals per year will have occasion to complete a survey at the conclusion of a TA engagement or event.

4. Planned Frequency of Information Collection

GDTA will collect information upon completion of each TA engagement or event, whether telephonic, web-based or in-person. GDTA will not conduct longitudinal follow-up (for example, to assess the longer-term impact of training).

5. Methods for Identifying Duplications

The information collected through this survey is unique to this program and not available elsewhere.

6. Time Period Over Which Information Will Be Collected

Data collection will occur from clearance of the instrument through the end of the contract’s performance period in 2016.

7. Expected Response Rate and Plan for Follow-up for Nonrespondents

A response rate of 40 to 60 percent is expected. GDTA will send three rounds of prompts to nonrespondents; the original round will immediately follow webinars, modules and virtual trainings (including telephonic TA), a reminder will be sent the following day, and a second reminder a week later. Follow ups for in-person events will occur three and five days following the events. Cross-referencing respondents against the universe of individuals to whom GDTA emailed the survey link will identify these nonrespondents. There will be no follow up for paper-based surveys.

8. Expected Ability to Assess Non-Response Bias Using Existing Information

A response rate of 40 to 60 percent is expected. GDTA does not expect nonresponse bias, but knowledge of some characteristics of nonrespondents based on pre-sample information will allow us to determine whether nonrespondents are distinct with regard to program experience, program type, or Center.

9. Methods Used to Maintain Customer Privacy

GDTA will submit the results of individual TA surveys to SAMHSA in aggregate, for example, via monthly reports. In addition, GDTA will not report individual respondents’ organizational affiliations to SAMHSA. In order to protect the confidentiality of responses for programs or Centers with only a single respondent, analytical results for programs or Centers with only a single potential respondent will be combined with those for other similar programs within the same Center; those that cannot be aggregated in this manner will have their individual program-specific results suppressed in reporting (with explanatory footnotes included in tables), but will be combined with other programs in the same Center, and with those with similar activity profile data using survey responses. In the same vein, GDTA may either aggregate all Center responses together or suppress the reporting output of results where presentation of data would allow a Center’s sole respondent’s answers to be identified. Any subsequent discussion of analysis “by program” should be understood to include such combinations or suppression of small-cell results.

D. Annual Response Burden Estimate

Training/TA Provider	Number of Respondents	Responses per Respondent	Total Number of Responses	Hours/ Response	Total Hours	Hourly Wage1	Total Hourly Cost
GDTA	500	1	500	0.1	50	$37.72	$1,886

¹ Hourly Wage: Given that most T/TA providers serve management-level personnel, the $37.72 rate is used as the mean hourly wage as reported on the Bureau of Labor Statistics website (http://www.bls.gov/). Annual hours are based on a 40-hour workweek for 48 weeks per year.

The estimated annualized cost to respondents for the proposed data collection activities is $1,886. For the purposes of estimating annual cost, GDTA assumes that participants will complete the satisfaction survey once.

Estimates of Annual Cost to the Government

The estimated annualized cost to the government for the proposed data collection activities is $80, which includes the expected cost for the Contracting Officer’s Representative (COR) overseeing the contract.

Position	Percent FTE	Annual Hours	Rate	Total Annual Cost
GDTA COR	0.1%	2	$40.00	$80.00
Totals		2		$80.00

E. Methods Used to Develop and Test the Questions

GDTA developed the survey questions based on standard instructional design and assessment practices in accordance with Kirkpatrick’s four-level evaluation model.

The common measures submitted here for OMB approval are the result of lengthy consultation and discussion among SAMHSA/GDTA personnel and training staff. SAMHSA/GDTA senior officials made the final selection of these measures.

F. Consultants within SAMHSA/GDTA and Outside the Agency

The consultant within SAMHSA for the survey is Dr. Darren Fulmore, who also serves as the Alternate Contracting Reporting Officer (ACOR) for the GDTA contract. Dr. Fulmore’s contact information is

Darren Fulmore, Ph.D.

Grantee Data Technical Assistance

Alternate Contracting Reporting Officer and CSAT Liaison

SAMHSA

1 Choke Cherry Road

Rockville, MD 20850

(240) 276-2824

[email protected]

The statistical consultant for the design is Dr. Steven Sullivan of The Cloudburst Group. Dr. Sullivan is an econometrician with a history of successful study design and implementation for SAMHSA, including the data collection and evaluation planning for the Co-occurring Disorders Integration and Innovation (CODI) contract. He is also the lead for the data collection efforts (using SurveyMonkey’s research.net service) and the analysis of satisfaction data from the survey. Dr. Sullivan’s contact information is

Steven T. Sullivan, Ph.D.

Senior Director and Subject Matter Expert

Cloudburst Consulting Group, Inc.

8400 Corporate Drive, Suite 550

Landover, MD 20785

(301) 385-6693

[email protected]

G. List of Attachments

Attachment A: GDTA Satisfaction Survey v3.0

Attachment B: Email to respondents

Attachment C: Screenshots of Survey

5