Justification

SAMHSA Customer Satisfaction Survey

Child, Adolescent and Family Branch

Technical Assistance Enterprise Feedback Surveys

A. Product/Activity to be Assessed

The Substance Abuse and Mental Health Services Administration’s (SAMHSA), Center for

Mental Health Services’ (CMHS) Child Adolescent and Family Branch (Branch) promotes and

ensures that the mental health needs of children and their families are addressed by a

community-based system of care. A component of the Branch’s work involves a range of activities designed to provide comprehensive technical assistance (TA) to CMHS system of care and expansion grantees. The feedback of these grantees is vital to the improvement of CMHS grantee services. For this reason, the Branch is requesting OMB approval for two customer satisfaction feedback surveys designed to provide input for one the Branch’s TA contractors:

• The Technical Assistance Site Visit Follow-Up Survey (Attachment 1) will assess grantee satisfaction with the contractor’s technical assistance delivered during an on-site visit. The survey will be administered following an on-site technical assistance visit. It will include fill-in-the-blank items and Likert scale items (strongly agree, agree, neutral, disagree, strongly disagree and n/a).

• The Technical Assistance Product Review Survey (Attachment 2) will assess grantee satisfaction with the written technical assistance products created by the contractor and accessed by grantees through the contractor’s website. The survey will be administered in an online format that website users can access following their experience with a product. It will include a fill-in-the-blank item and Likert scale items (not useful at all, somewhat useful, very useful; not applicable at all, somewhat applicable, very applicable; no, maybe, yes).

B. Brief Statement of Objectives

These surveys will assess participant satisfaction with the contractor’s technical assistance services and products. The data collected from the evaluations will include the following components:

• Technical Assistance Site Visit Follow-Up Survey

• Specific satisfaction – responses will determine mean satisfaction across multiple items, indicating the extent to which participants are satisfied with aspects of the on-site visit.

• Personal satisfaction – participants’ comments and recommendations will provide rich, contextualized information.

• The Branch and the contractor will use the data gathered by the survey to continuously improve TA services. To analyze the data, the contractor will compute descriptive statistics (e.g., mean, median, and frequency) for the Likert scale and summarize the open-ended comments.

• Technical Assistance Product Review Survey

  • Specific satisfaction – responses will determine mean satisfaction across multiple items, indicating the extent to which participants are satisfied with aspects of the technical assistance products.

  • Personal satisfaction – participants’ comments and recommendations will provide rich, contextualized information about the products.

  • The Branch and the contractor will use the data gathered by the survey to continuously improve TA products. To analyze the data, the contractor will compute descriptive statistics (e.g., mean, median, and frequency) for the Likert scale and summarize the open-ended comments.

C. Overview of Methods to Collect the Information

Data collection method.

• Technical Assistance Site Visit Follow-Up Survey: The contractor will administer the Technical Assistance Site Visit Follow-Up Survey electronically using Survey Monkey. Grantee communities will receive an email requesting their completion of the survey and an electronic link to it in Survey Monkey. The survey uses commonly asked questions to solicit feedback regarding participant satisfaction with TA services.

• Technical Assistance Product Review Survey: The contractor will administer the Technical Assistance Product Review Survey electronically using a survey embedded in the contractor’s website. While accessing products electronically, grantee communities will have the opportunity to access and complete the survey. The survey uses commonly asked questions to solicit feedback regarding grantee satisfaction with TA products.

Method for identifying respondents.

• Technical Assistance Site Visit Follow-Up Survey: The names and contact information of participants in the on-site technical assistance visit will be collected. All participants will be eligible and encouraged to complete this survey.

• Technical Assistance Product Review Survey: All individuals who access the technical assistance products via the website will be encouraged to complete the survey. Anyone who accesses technical assistance products will be eligible to complete the survey.

Proposed sample size and rationale.

• Technical Assistance Site Visit Follow-Up Survey: The contractor estimates that approximately 50 individuals from the Expansion Jurisdictions will receive this survey. The contractor will ask all of these individuals to complete this survey, to help obtain representative feedback of all community members. Assuming that 35% of these individuals complete this survey, 17 participants in on-site visits for each jurisdiction will complete this survey which is administered after the on-site visit.

• Technical Assistance Product Review Survey: Based on the contractor’s web analytics data, the contractor estimates that approximately 2,400 unique individuals visit the contractor’s website monthly. All of these individuals will be able to complete this survey. Assuming that 2% of these individuals complete this survey, 48 website visitors will complete this survey monthly.

Planned frequency of information collection.

• Technical Assistance Site Visit Follow-Up Survey: The contractor will administer the Technical Assistance Site Visit Follow-Up Survey after each on-site technical assistance visit.

• Technical Assistance Product Review Survey: The message encouraging individuals to complete the Product Review Survey and the link to the survey will be consistently available for website users to access.

Time period over which the information will be collected.

• Technical Assistance Site Visit Follow-Up Survey: This survey will remain active for approximately four weeks.

• Technical Assistance Product Review Survey: This survey will remain active for the foreseeable future, until the decision is made to remove the product from the site or to terminate data collection.

Expected response rate and plan for follow-up, if any, of non-respondents.

• Technical Assistance Site Visit Follow-Up Survey: Based on past experience collecting similar feedback, the contractor anticipates that approximately 35% of grantee community members will complete the survey. The contractor will send to grantee community members an email with a Survey Monkey link to the survey. Within one week of the survey due date, the contractor will send an email asking grantee community members to complete the survey. This email will again include a survey link. The contractor will send an additional reminder email to all potential respondents around the survey due date. No further follow-up activities are proposed.

• Technical Assistance Product Review Survey: The contractor estimates that approximately 2% of website visitors will complete the survey. The contractor will post the survey on the contractor’s website with text that encourages website users to complete it. There will be no follow-up with respondents, unless specifically requested in the open-ended comments.

Expected ability to assess non-response bias using existing information.

• Technical Assistance Site Visit Follow-Up Survey: The contractor does not anticipate any non-response bias based on existing information.

• Technical Assistance Product Review Survey: The contractor does not anticipate any non-response bias based on existing information.

Methods used to maintain customer privacy.

• Technical Assistance Site Visit Follow-Up Survey: Responses will be anonymous. The introductory text for the survey explains this for respondents. Respondents will reply to the surveys using a common survey link, so it will not be possible to identify respondents unless they self-identify in the open-ended comments. Since we use the same link for everyone, we assume the possibility of duplicate responses from the same person. We don’t anticipate duplicate responses, though.

• Technical Assistance Product Review Survey: Responses will be anonymous. The introductory text for the survey explains this for respondents. Respondents will reply to the surveys using a common survey link, so it will not be possible to identify respondents unless they self-identify in the open-ended comments. Since we use the same link for everyone, we assume the possibility of duplicate responses from the same person. We don’t anticipate duplicate responses, though.

D. Annual Response Burden Estimate

The following table provides information about the annual response burden estimated for these consumer feedback surveys.

Respondent	Number of Respondents (a)	Number of Responses/ Respondent (b)	Hours/ Response in Minutes (c)	Annual Burden Hours	Hourly Wage (d)	Annual Cost ($)
Technical Assistance Site Visit Follow-Up Survey	50	1	0.20	10	21.07	211
Technical Assistance Product Review Survey	576	1	0.20	115	21.07	2,423
Total	626			125		2,634

(a) The number of respondents has been estimated using the average number of participants in on-site technical assistance visits to date. For the product review survey, the number of respondents is estimated for a 12-month period (48 respondents monthly).

(b) For the purposes of estimating annual cost, the contractor assumes that community members will complete each survey only once

(c) The average burden per response was estimated based on independent review of the

surveys by contractor staff.

(d) The mean hourly wage is estimated at $21.07 for the types of respondents based on a mean

yearly salary of $43,830 (assessed through a review of 2011 average salary estimates of community and social service occupations from the US Department of Labor’s Bureau of Labor Statistics, at http://www.bls.gov/oes/current/oes_nat.htm).

E. Methods Used to Develop and Test the Questions

• Technical Assistance Site Visit Follow-Up Survey: The contractor prepared the consumer feedback survey. The contactor circulated several draft versions that were critiqued by the contractor’s management team and other staff for appropriate content and clarity of questions.

• Technical Assistance Product Review Survey: The contractor prepared the consumer feedback survey. The contractor circulated several draft versions that were critiqued by the contractor’s management team and other staff for appropriate content and clarity of questions.

F. Federal Project Officer and Contractor

Diane Sondheimer

Federal Project Officer

SAMHSA/CMHS

Child Adolescent and Family Branch

1 Choke Cherry Road, Room 6-1048

Rockville, MD 20857

Phone: (240) 276-1922

Fax: 240.276.1990

Email: [email protected]

Regenia Hicks

Project Director

American Institutes for Research

1000 Thomas Jefferson St., NW

Washington, DC 20007

Phone: 202.403.5381

Fax: 202.403.5007

Email: [email protected]

Jeffrey Poirier

Senior Researcher

American Institutes for Research

1000 Thomas Jefferson St., NW

Washington, DC 20007

Phone: 202.403.5368/202.329.5440

Fax: 202.403.5007

Email: [email protected]

G. Project Statistician and Data Collection Entity

• Technical Assistance Site Visit Follow-Up Survey: The contractor will collect and compile data using Survey Monkey. The use of compiled and analyzed results is at the sole discretion of the Branch.

• Technical Assistance Product Review Survey: The contractor will collect and compile data using a database management system embedded in the contractor’s website. The use of compiled and analyzed results is at the sole discretion of the Branch.

List of Attachments

Attachment 1: Technical Assistance Site Visit Follow-Up Survey

Attachment 2: Technical Assistance Product Review Survey

5