Supporting Statement

SAMHSA Customer Satisfaction Survey

Center for Behavioral Health Statistics and Quality

Grantee Data Technical Assistance (GDTA) Learning Community Satisfaction Survey

A. Product/Activity to be Assessed

The Substance Abuse and Mental Health Services Administration’s (SAMHSA’s) Grantee Data Technical Assistance (GDTA) contract provides technical assistance (TA) for SAMHSA discretionary services grantees and project officers on data collection, management, and utilization. Designed to improve the accuracy, completeness, and timeliness of grantee data, GDTA promotes the use of data to improve practice, program design, and policy development. SAMHSA is requesting approval from the Office of Management and Budget (OMB) to implement a satisfaction survey for participants in Learning Community (LC) events supported by the GDTA contract. GDTA considers it important to assess participants’ perception of the value of services provided in these LC events.

B. Brief Statement of Objectives

In 2014, the Center for Behavioral Health Statistics and Quality (CBHSQ) funded the GDTA contract to provide training and TA to all grantees receiving funding from the Center for Substance Abuse Treatment (CSAT), the Center for Mental Health Services (CMHS), and some grantees receiving funding from the Center for Substance Abuse Prevention (CSAP). GDTA Learning Communities will focus on helping grantees use their GPRA data for performance management and monitoring, as well as services improvement. The information being collected in this survey will inform CHBSQ regarding whether the Learning Communities are meeting their objectives and if participants are finding the Learning Communities useful.

Approval of this survey and associated data collection activities will allow SAMHSA to continue to support its GPRA reporting requirements that quantify the effects and accomplishments of its grant programs. As the GDTA contract supports grantees in using their GPRA data, this data collection will ultimately assist SAMHSA’s legislative mandate to increase access to high quality prevention and treatment services and to improve outcomes. All of SAMHSA’s programs and activities are geared toward the achievement of goals related to reducing the impact of mental illness and addiction. GPRA performance monitoring, and training on that monitoring, are collaborative and cooperative aspects of this process.

C. Overview of Methods to Collect Information

1. Data Collection Method

CBHSQ’s mission, through the GDTA contract, is to provide SAMHSA grantees with the most informative and useful training and TA possible, in order for grantees to use their GPRA data to improve their performance. To determine what is informative and useful, CBHSQ will conduct an eight-question Learning Community Survey of SAMHSA-funded grantees who are participating in GDTA Learning Communities. Data collected from this Learning Community Survey will be used by CBHSQ and the GDTA contract to continually improve and tailor the Learning Communities.

Surveys will include eight questions. The first question asks respondents to identify the areas in which they increased their understanding. The second question asks which Learning Community focus areas the respondent participated in: building a culture of evidence; institutionalizing performance improvement with data; data collection and data quality standards; data integration and data reporting; and data use and data visualization. Questions 3-7 ask about the objectives of the Learning Communities; participants will only complete the questions for the Learning Communities that they indicated in Question 2. Questions 3-7 read: “As a result of participating in GDTA activities over the past six months, has your organization…” followed by the specific learning objectives for that topic area (e.g., “Increased its capacity to measure the relative impact of agency/program services?”). The response options are “yes,” “no,” and “don’t know/not sure.” The final question asks respondents to indicate their agreement or disagreement with three statements regarding the Learning Communities (e.g., “The Learning Community presented ideas that are useful to me in my current job.”). Response options consist of five scoring elements with a Likert scale from “strongly agree” to “strongly disagree.”

2. Method for Identifying Respondents

Respondents are limited to SAMHSA-funded grantees who are participating in GDTA Learning Communities for at least six months. The organization of the Learning Communities permits identification of members and therefore the distribution of survey materials via their contact information.

Respondents will receive an email from the GDTA contractor that invites them to participate and includes a link to the survey [Attachment B]. When respondents have completed the survey, they will electronically submit their answers in SurveyMonkey (http://www.surveymonkey.com). SAMHSA expects almost all responses to be submitted electronically. There will be no requirements for respondents to maintain or save any data onto their local computers.

3. Proposed Sample Size and Rationale

The GDTA proposes to collect data from up to 250 Learning Community participants at six-month intervals. This is based on the anticipated number of participants and the need to collect periodic follow-up information.

4. Planned Frequency of Information Collection

The Learning Community Survey is voluntary, and each respondent is asked to respond to the survey only after a minimum of six months of participation in Learning Community events. Individual respondents should expect to be contacted no more than twice per year for each Learning Community. If these data are not collected, the GDTA contract will not be able to create targeted training and TA that is the most useful and informative to grantees. If these data are not collected continually as members receive training and TA by joining and participating in Learning Communities, the GDTA contract may produce training and TA products that are not on-target with the grantees’ current needs.

5. Methods for Identifying Duplications

The information collected through this survey is unique to this program and not available elsewhere.

6. Time Period Over Which Information Will Be Collected

Data will be collected following the clearance of the instrument, starting no sooner than six months following the establishment of the first Learning Community and continue throughout 2016.

7. Expected Response Rate and Plan for Follow-up of Nonrespondents

A response rate of 40 to 60 percent is expected. GDTA’s approach to maximizing response rates is to provide two reminders to respondents throughout the time allotted for answering the survey (anticipated to be one month). SAMHSA is employing the Learning Community member roster as an outreach database with contact information for each participant, as well as event information, in order to email and partially pre-populate the surveys. SAMHSA will use the same database (cross-referenced against data on returned surveys in research.net) to email reminders one week and four weeks after initial distribution for any outstanding surveys.

8. Expected Ability to Assess Non-Response Bias Using Existing Information

It is the GDTA’s current expectation that applications of the Learning Community Survey will not be amenable to an analysis of potential nonresponse bias unless Learning Community membership is large, defined as containing 100 or more members in an individual group.

The primary analysis of nonresponse bias would take the form of either chi-squared analyses or logistic regressions to determine whether nonresponse is statistically independent of the characteristics of grantees or SAMHSA staff that are observable at the sampling stage. In the regression specifications, an indicator variable for nonresponse would serve as the dependent variable.

9. Methods Used to Maintain Customer Privacy

This data collection will not include personal identifiers. As previously mentioned, SAMHSA will use the email addresses of grantees to distribute the surveys and pre-populate answers such as event title and type, but this information will not be included in the resulting analytical data set. Other information in the database, such as Center and program, will be used for analysis but will not be used to identify individual respondents.

D. Annual Response Burden Estimate

CBHSQ expects the universe of respondents to be Learning Community event attendees from every SAMHSA-funded grant that will be served by the GDTA contract, as well as SAMHSA staff who will also be served by TA and training. GDTA has estimated 250 participants responding twice at six-month intervals, resulting in a total of 500 responses. The average estimated response time per participant is 10 minutes, which is 0.17 hours.

Survey Name	Number of Respondents	Responses per Respondent	Total Number of Responses	Hours per Response	Total Annual Hour Burden	Hourly Wage Cost	Total Annual Hour Cost ($)
Grantee Learning Community Survey	250	2	500	0.17	85	$37.72	$3,206

Notes: Hourly wage cost estimates are based on the Bureau of Labor Statistics “employer costs per hour worked for employee compensation and costs for state and government workers” for June 2014. Grantee staff from clerical/data entry professionals to project directors, as well as SAMHSA professional staff, may be asked to complete the survey; therefore the “management, professional, and related” occupational group was used to produce a conservative burden estimate. These wages are fully loaded. Accessible here: http://www.bls.gov/news.release/ecec.t03.htm

Although the grantees and staff being surveyed vary greatly in their program focus and organizational structure, it is not anticipated that the hour burden on respondents will vary by respondent. There are no direct costs to respondents other than their time to participate. There is no recordkeeping burden to respondents for this data collection effort.

E. Methods Used to Develop and Test the Questions

GDTA developed the survey questions based on standard instructional design and assessment practices in accordance with Kirkpatrick’s four-level evaluation model.

The common measures submitted here for OMB approval are the result of lengthy consultation and discussion among SAMHSA/GDTA personnel and training staff. SAMHSA/GDTA senior officials made the final selection of these measures.

F. Consultants within SAMHSA/GDTA and Outside the Agency

The consultant within SAMHSA for the survey is Dr. Darren Fulmore, who also serves as the Alternate Contracting Reporting Officer (ACOR) for the GDTA contract. Dr. Fulmore’s contact information is

Darren Fulmore, Ph.D.

Grantee Data Technical Assistance

Alternate Contracting Reporting Officer and CSAT Liaison

SAMHSA

1 Choke Cherry Road

Rockville, MD 20850

(240) 276-2824

[email protected]

The statistical consultant for the design is Dr. Steven Sullivan of The Cloudburst Group. Dr. Sullivan is an econometrician with a history of successful study design and implementation for SAMHSA, including the data collection and evaluation planning for the Co-occurring Disorders Integration and Innovation (CODI) contract. He is also the lead for the data collection efforts (using SurveyMonkey’s research.net service) and the analysis of satisfaction data from the survey. Dr. Sullivan’s contact information is

Steven T. Sullivan, Ph.D.

Senior Director and Subject Matter Expert

Cloudburst Consulting Group, Inc.

8400 Corporate Drive, Suite 550

Landover, MD 20785

(301) 385-6693

[email protected]

G. List of Attachments

Attachment A: GDTA Learning Community Survey

Attachment B: Email to respondents

Attachment C: Screenshots of Survey

5