Justification

SAMHSA Customer Satisfaction Survey
Child, Adolescent and Family Branch
Learning Event Feedback Survey
A.

Product/Activity to be Assessed

The Substance Abuse and Mental Health Services Administration’s (SAMHSA), Center for
Mental Health Services’ (CMHS) Child Adolescent and Family Branch (Branch) promotes and
ensures that the mental health needs of children and their families are addressed by a
community-based system of care. A component of the Branch’s work involves hosting on-site
learning events in which the contractor delivers information about providing culturally and
linguistically competent services to lesbian, gay, bisexual, transgender, questioning, intersex, and
two-spirit (LGBTQI2-S) children, families, and youth. The feedback from these learning events
is vital to the improvement of service to system of care grantees. For this reason, the contractor is
requesting OMB approval for a customer satisfaction feedback survey designed to collect
customer satisfaction feedback on learning and technical assistance events.
The On-Site Learning Event Feedback Form (Attachment 1) and Follow-Up Learning Event
Feedback Form (Attachment 2) will assess grantee satisfaction with the subcontractor’s
learning events following the learning events. It will include primarily Likert scale items
(strongly disagree, disagree, agree, strongly agree, not applicable) and three open-ended
questions. The contractor will administer the On-Site Learning Event Feedback Form at the
time of a learning event; the contractor will administer the Follow-Up Learning Event
Feedback Form 6-9 months following a learning event. The follow-up administration is
important for accessing how satisfied participants are 6-9 months after the learning event, how
they have used the information shared, and what training needs they still have.
B.

Brief Statement of Objectives

The survey will assess participant satisfaction with the contractor’s TA services. The data
collected from the evaluations will include the following components:
o Specific satisfaction – responses will determine mean satisfaction across multiple
items, indicating the extent to which participants are satisfied with the learning
event content and the content on the LGBTQI2-S toolkit.
o Personal satisfaction – participants’ comments and recommendations will
provide rich, contextualized information on the learning event.
The Branch and the contractor will use the data gathered by the survey to continuously improve
TA services and to better assess the needs of grantees as part of future TA efforts. To analyze
the data, the contractor will compute descriptive statistics (e.g., mean, median, and frequency)
for the Likert scale items and summarize the open-ended comments.
C.

Overview of Methods to Collect the Information

Data collection method.
 The contractor will administer the learning event feedback form by paper at the end of
learning events and again electronically at some time following the learning events (e.g.,
1

6 or 9 months) using Survey Monkey. The form will be shared in person following
learning events. The survey follow-up will include an email requesting their completion
of the survey with an electronic link to it in Survey Monkey. The survey uses targeted
questions to solicit feedback regarding participant satisfaction with the learning events.
Method for identifying respondents.
 All grantee community members who participate in the learning events will be eligible
and encouraged to complete this feedback form.
Proposed sample size and rationale.
 Approximately 400 individuals attended learning events in September 2011, so an
electronic follow-up feedback form will be sent to them. The contractor will ask all of
these individuals to complete this form, to help obtain representative feedback of all
community members. Assuming that 20% of these individuals complete this survey, 80
grantee community members will complete the survey.
 The contractor estimates that 100 individuals will participate in future learning events.
Approximately 90 of these individuals will complete paper feedback forms on site at the
time of the learning events. Approximately 20 of these individuals will also complete an
electronic follow-up survey 6-9 months after the learning event.
Planned frequency of information collection.
 The contractor will administer the feedback form twice, once at the time of the learning
event and 6-9 months following the learning event to assess ongoing satisfaction with the
technical assistance.
Time period over which the information will be collected.
 The electronic survey will remain active for approximately four weeks.
Expected response rate and plan for follow-up, if any, of non-respondents.
 Based on past experience collecting similar feedback, the contractor anticipates that
approximately 90% of grantee community members will complete the on-site feedback
form (at the time of the learning event) and 20% will complete the electronic follow-up
form (administered 6-9 months after the learning event).
 For the follow-up feedback form, the contractor will send to grantee community members
an email with a Survey Monkey link to the survey. Within one week of the survey due
date, the contractor will send an email asking grantee community members to complete
the survey. This email will again include a survey link. The contractor will send an
additional reminder email to all potential respondents around the survey due date. No
further follow-up activities are proposed.
Expected ability to assess non-response bias using existing information.
 The contractor does not anticipate any non-response bias based on existing information.
Methods used to maintain customer privacy.
 Responses will be anonymous unless respondents self-identify themselves in the openended responses. For the on-site administration, the contractor will ask participants to
return completed surveys in a pile so that responses cannot be associated with particular
individuals.
2



D.

Follow-up respondents will reply to the surveys using a common survey link, so it will
not be possible to identify respondents unless they self-identify in the open-ended
comments. Since the contractor will use the same link for everyone, the contractor
assumes the possibility of duplicate responses from the same person. The contractor does
not anticipate duplicate responses, though.
Annual Response Burden Estimate
The following table provides information about the annual response burden estimated for
this learning event feedback form.

Number of
Respondents
(a)

Number of
Responses/
Respondent
(b)

Hours/
Response in
Minutes (c)

Annual
Burden
Hours

Hourly
Wage (d)

Annual
Cost ($)

Grantee Communities (on-site
administration)

90

1

0.15

13.5

21.07

284

Grantee Communities (followup administration)

100

1

0.15

15.0

21.07

316

Total

190

1

0.15

28.5

21.07

600

Respondent

(a) The number of respondents has been estimated using the number of learning event
participants and past response rates.
On-site administration: The contractor estimates that 100 individuals will participate in future
learning events. Approximately 90 of these individuals will complete paper feedback forms
on site at the time of the learning events.
Follow-up administration: Approximately 400 individuals attended learning events in
September 2011, so an electronic follow-up feedback form will be sent to them. The
contractor will ask all of these individuals to complete this form, to help obtain representative
feedback of all community members. Assuming that 20% of these individuals complete this
survey, 80 grantee community members will complete the survey. In addition, approximately
20 of the individuals who participate in future learning events will also complete an
electronic follow-up survey 6-9 months after these future learning events. This equates to
approximately 100 total respondents (80+20).
(b) For the purposes of estimating annual cost, the contractor assumes that community members
will complete each survey only once.

(c) The average burden per response was estimated based on independent review of the
surveys by contractor staff.
(d) The mean hourly wage is estimated at $21.07 for the types of respondents based on a mean
yearly salary of $43,830 (assessed through a review of 2011 average salary estimates of
community and social service occupations from the US Department of Labor’s Bureau of Labor
Statistics, at http://www.bls.gov/oes/current/oes_nat.htm).

3

E.

Methods Used to Develop and Test the Questions

The contractor prepared the consumer feedback survey. The contactor circulated several draft
versions and critiqued by the contractor’s management team and other staff for appropriate
content and clarity of questions.
F.

Federal Project Officer and Contractor
Susan Stromberg
Federal Project Officer
SAMHSA/CMHS
Child Adolescent and Family Branch
1 Choke Cherry Road, Room 6-1048
Rockville, MD 20857
Phone: 240.276.1929
Fax: 240.276.1990
Email: [email protected]
Regenia Hicks
Project Director
American Institutes for Research
1000 Thomas Jefferson St., NW
Washington, DC 20007
Phone: 202.403.5381
Fax: 202.403.5007
Email: [email protected]
Jeffrey Poirier
Senior Researcher
American Institutes for Research
1000 Thomas Jefferson St., NW
Washington, DC 20007
Phone: 202.403.5368/202.329.5440
Fax: 202.403.5007
Email: [email protected]

G.

Project Statistician and Data Collection Entity

The contractor will collect and compile data on site using a paper survey; for the follow-up
survey, the contractor will administer this electronically using Survey Monkey. The use of
compiled and analyzed results is at the sole discretion of the Branch, but the contractor
anticipates data will be shared with the system of care communities hosting the learning events
since they are also interested in reviewing participant feedback.

List of Attachments
Attachment 1: On-Site Learning Event Feedback Form
Attachment 2: Follow-Up Learning Event Feedback Form
4