Public Engagement Platform Surveys

SAMHSA Customer Satisfaction Survey

Office of Communications, Public Engagement Platform

Customer Feedback Surveys

A. Product/Activity to be Assessed

The Substance Abuse and Mental Health Services Administration’s (SAMHSA) Office of Communications (OC) Public Engagement Platform (PEP) is committed to expanding the availability of SAMHSA’s content for health professionals and the public, and enabling SAMHSA’s key messages and evidence-based materials to reach existing and new audiences. In alignment with the Federal Digital Strategy, SAMHSA OC is participating in an ongoing effort to make a growing proportion of its products available through an array of digital dissemination channels, such as mobile applications (apps)—namely, the Bullying Prevention App and Suicide Prevention App. In addition, SAMHSA’s 24/7 Contact Center remains a main point of entry for both behavioral health professionals and consumers to SAMHSA’s services and publications. This entry point provides SAMHSA with an important feedback opportunity for its audiences. Feedback from these audiences is essential to enhancing the quality and utility of SAMHSA’s products and services. For this reason, SAMHSA’s OC is requesting Office of Management and Budget approval for the following three customer satisfaction feedback surveys:

• The SAMHSA Bullying Prevention App: User Feedback Survey (Attachment 1) will assess users’ satisfaction with the content and functionality of the Bullying Prevention App. A link to a web-based survey will be embedded directly within the mobile app. It will include multiple choice questions, fill-in-the-blank items, and Likert scale items (extremely likely, likely, neutral, unlikely, not at all likely).

• The SAMHSA Suicide Prevention App: User Feedback Survey (Attachment 2) will assess users’ satisfaction with the content and functionality of the Suicide Prevention App. A link to a web-based survey will be embedded directly within the mobile app. It will include multiple choice questions, fill-in-the-blank items, and Likert scale items (extremely likely, likely, neutral, unlikely, not at all likely).

• The SAMHSA Contact Center Customer Satisfaction Survey (Attachment 3) will assess users’ satisfaction with the delivery of services and resources through the Contact Center. Participants will be invited by the information specialist to participate in the survey at the end of a call for general information or a publication request, or via email in the signature line of the [email protected] email address. The survey will be administered by telephone via interactive voice response (IVR) or as a web-based form for email customers. It will include multiple choice questions and Likert scale items (extremely likely, likely, neutral, unlikely, not at all likely).

B. Brief Statement of Objectives

These surveys will assess participant satisfaction with SAMHSA’s services and products. The data collected from the evaluations will include the following components:

• SAMHSA Bullying Prevention App: User Feedback Survey

  • Specific satisfaction – responses will determine mean satisfaction across multiple items, indicating the extent to which participants are satisfied with aspects of the Bullying Prevention App.

  • Personal satisfaction – participants’ comments and recommendations will provide rich, contextualized information about personal app usage.

  • OC and the contractor will use the data gathered by the survey to assess user demographics, determine the nature of app usage, and assess priorities for future app enhancements. To analyze the data, the contractor will compute descriptive statistics (e.g., weighted mean and frequency) for the Likert scale and summarize the open-ended comments.

• SAMHSA Suicide Prevention App: User Feedback Survey

  • Specific satisfaction – responses will determine mean satisfaction across multiple items, indicating the extent to which participants are satisfied with aspects of the Suicide Prevention App.

  • Personal satisfaction – participants’ comments and recommendations will provide rich, contextualized information about personal app usage.

  • OC and the contractor will use the data gathered by the survey to assess user demographics, determine the nature of app usage, and assess priorities for future app enhancements. To analyze the data, the contractor will compute descriptive statistics (e.g., weighted mean and frequency) for the Likert scale and summarize the open-ended comments.

• SAMHSA Contact Center Customer Satisfaction Survey

  • Specific satisfaction – responses will determine mean satisfaction across multiple items, indicating the extent to which participants are satisfied with the delivery of services and resources through the Contact Center.

  • Personal satisfaction – participants’ comments and recommendations will provide rich, contextualized information.

  • OC and the contractor will use the data gathered by the survey to assess customer demographics, satisfaction with SAMHSA’s services, and priorities for future service. To analyze the data, the contractor will compute descriptive statistics (e.g., weighted mean and frequency) for the Likert scale and summarize the open-ended comments.

C. Overview of Methods to Collect the Information

Data collection method.

• SAMHSA Bullying Prevention App: User Feedback Survey: The contractor will administer the SAMHSA Bullying Prevention App: User Feedback Survey electronically using a web-based survey. Mobile app users can access the embedded survey within the app. The survey uses commonly asked questions to solicit feedback regarding participant satisfaction with the mobile app.

• SAMHSA Suicide Prevention App: User Feedback Survey: The contractor will administer the SAMHSA Suicide Prevention App: User Feedback Survey electronically using a web-based survey. Mobile app users can access the embedded survey within the app. The survey uses commonly asked questions to solicit feedback regarding participant satisfaction with the mobile app.

• SAMHSA Contact Center Customer Satisfaction Survey: The contractor’s information specialists will invite customers to participate in the survey at the end of their telephone or email transaction. The data collection will occur electronically via an IVR-based survey for callers or in a web-based survey for email customers. The survey uses commonly asked questions to solicit feedback regarding customer satisfaction with services and products.

Method for identifying respondents.

• SAMHSA Bullying Prevention App: User Feedback Survey: All individuals who access the Bullying Prevention App will be encouraged to complete the survey. Anyone who accesses the app will be eligible to complete the survey.

• SAMHSA Suicide Prevention App: User Feedback Survey: All individuals who access the Suicide Prevention App will be encouraged to complete the survey. Anyone who accesses the app will be eligible to complete the survey.

• SAMHSA Contact Center Customer Satisfaction Survey: All individuals who access the SAMHSA Contact Center for general inquiries or publication requests—via telephone or email—will be encouraged to complete the survey. Anyone who accesses the Contact Center with general questions and publication orders will be eligible to complete the survey.

Proposed sample size and rationale.

• SAMHSA Bullying Prevention App: User Feedback Survey: Based on the contractor’s experience with similar projects, the contractor estimates that approximately 1,750 unique individuals will use the contractor’s mobile app monthly. All of these individuals will be able to complete this survey. Assuming that 1 percent of these individuals complete this survey, 18 mobile app users will complete this survey monthly.

• SAMHSA Suicide Prevention App: User Feedback Survey: Based on the contractor’s experience with similar projects, the contractor estimates that approximately 1,750 unique individuals will use the contractor’s mobile app monthly. All of these individuals will be able to complete this survey. Assuming that 1 percent of these individuals complete this survey, 18 mobile app users will complete this survey monthly.

• SAMHSA Contact Center Customer Satisfaction Survey: The contractor receives an average of 3,800 general information and publication inquiries through the Contact Center per month. Assuming that 1 percent of these individuals complete this survey, 38 Contact Center customers will complete this survey monthly.

Planned frequency of information collection.

• SAMHSA Bullying Prevention App: User Feedback Survey: The message encouraging individuals to complete the User Feedback Survey and the embedded link to the online survey will be consistently available for app users to access.

• SAMHSA Suicide Prevention App: User Feedback Survey: The message encouraging individuals to complete the User Feedback Survey and the embedded link to the online survey will be consistently available for app users to access.

• SAMHSA Contact Center Customer Satisfaction Survey: Invitations to complete the survey—either from the information specialist or email signature line—will be consistently delivered to recipients of Contact Center products or services.

Time period over which the information will be collected.

• SAMHSA Bullying Prevention App: User Feedback Survey: This survey will remain active for the foreseeable future, until the decision is made to remove the product from the site or to terminate data collection.

• SAMHSA Suicide Prevention App: User Feedback Survey: This survey will remain active for the foreseeable future, until the decision is made to remove the product from the site or to terminate data collection.

• SAMHSA Contact Center Customer Satisfaction Survey: This survey will remain active for the foreseeable future, until the decision is made to terminate data collection.

Expected response rate and plan for follow-up, if any, of nonrespondents.

• SAMHSA Bullying Prevention App: User Feedback Survey: The contractor estimates that approximately 1 percent of app users will complete the survey. The contractor will embed a link to the survey within the app with text that encourages users to complete it. There will be no follow-up with respondents, unless specifically requested in the open-ended comments.

• SAMHSA Suicide Prevention App: User Feedback Survey: The contractor estimates that approximately 1 percent of app users will complete the survey. The contractor will embed a link to the survey within the app with text that encourages website users to complete it. There will be no follow-up with respondents, unless specifically requested in the open-ended comments.

• SAMHSA Contact Center Customer Satisfaction Survey: The contractor estimates that approximately 1 percent of Contact Center customers with general inquiries or publication orders will complete the survey. The contractor staff will encourage customers at the end of a phone call or email transaction to complete the survey. There will be no follow-up with respondents, unless specifically requested in the open-ended comments.

Expected ability to assess nonresponse bias using existing information.

• SAMHSA Bullying Prevention App: User Feedback Survey: The contractor does not anticipate any nonresponse bias based on existing information.

• SAMHSA Suicide Prevention App: User Feedback Survey: The contractor does not anticipate any nonresponse bias based on existing information.

• SAMHSA Contact Center Customer Satisfaction Survey: The contractor does not anticipate any nonresponse bias based on existing information.

Methods used to maintain customer privacy.

• SAMHSA Bullying Prevention App: User Feedback Survey: Responses will be anonymous. The introductory text for the survey explains this for respondents. Respondents will reply to the surveys using a common survey link, so it will not be possible to identify respondents unless they self-identify in the open-ended comments. Since we use the same link for everyone, we assume the possibility of duplicate responses from the same person. However, we do not anticipate duplicate responses.

• SAMHSA Suicide Prevention App: User Feedback Survey: Responses will be anonymous. The introductory text for the survey explains this for respondents. Respondents will reply to the surveys using a common survey link, so it will not be possible to identify respondents unless they self-identify in the open-ended comments. Since we use the same link for everyone, we assume the possibility of duplicate responses from the same person. However, we do not anticipate duplicate responses.

• SAMHSA Contact Center Customer Satisfaction Survey: Responses will be anonymous. The introductory transcript for the survey explains this for respondents. Respondents will reply to the surveys using an IVR application if completing the survey over the phone, or using a common link if responding through the link in the email signature line, so it will not be possible to identify respondents. Since we use the same app for everyone, we assume the possibility of duplicate responses from the same person. However, we do not anticipate duplicate responses.

D. Annual Response Burden Estimate

The following table provides information about the annual response burden estimated for these consumer feedback surveys.

Respondent	Number of Respondents1	Number of Responses/ Respondent2	Hours/ Response in Minutes3	Annual Burden Hours	Hourly Wage4	Annual Cost ($)
SAMHSA Bullying Prevention App: User Feedback Survey	216	1	0.08	17.3	$22.33	386
SAMHSA Suicide Prevention App: User Feedback Survey	216	1	0.08	17.3	$71.90	1,244
SAMHSA Contact Center Customer Satisfaction Survey	456	1	0.05	22.8	$28.79	656
Total	888			57.4		2,286

¹ For the mobile apps, the number of respondents has been estimated using projections for app usage. For the Contact Center, the number of respondents has been estimated based on actual call volume over the prior 12 months.

² For the purposes of estimating annual cost, the contractor assumes that community members will complete each survey only once.

³ The average burden per response was estimated based on independent review of the surveys by contractor staff.

⁴ Contractor staff reviewed 2013 average salary estimates of occupations from the U.S. Department of Labor’s Bureau of Labor Statistics, at http://www.bls.gov/oes/current/oes_nat.htm, to calculate the mean hourly wage estimates. The mean hourly wage is estimated at $22.33 for Bullying Prevention App respondents based on a mean yearly salary of $46,440. The mean hourly wage is estimated at $71.90 for Suicide Prevention App respondents based on a mean yearly salary of $149,559. The mean hourly wage is estimated at $28.79 for Contact Center respondents based on a mean yearly salary of $59,921.

E. Methods Used To Develop and Test the Survey

• SAMHSA Bullying Prevention App: User Feedback Survey: The contractor prepared the consumer feedback survey. The contractor circulated several draft versions that were critiqued by the contractor’s management team and other staff for appropriate content and clarity of the survey.

• SAMHSA Suicide Prevention App: User Feedback Survey: The contractor prepared the consumer feedback survey. The contractor circulated several draft versions that were critiqued by the contractor’s management team and other staff for appropriate content and clarity of the survey.

• SAMHSA Contact Center Customer Satisfaction Survey: The contractor prepared the consumer feedback survey. The contractor circulated several draft versions that were critiqued by the contractor’s management team and other staff for appropriate content and clarity of the survey.

F. Federal Project Officer and Contractor

Anne Mathews-Younes

Federal Project Officer

SAMHSA/OA/OC

1 Choke Cherry Road, Room 6-1048

Rockville, MD 20857

Phone: (240) 276-1837

Fax: (240) 276-1870

Email: [email protected]

Ileana Quintas

Project Director and CEO

IQ Solutions, Inc.

11300 Rockville Pike, Suite 901

Rockville, MD 20852

Phone: (240) 221-4230

Fax: (240) 984-1473

Email: [email protected]

Stephanie Adams

Vice President, Health Intelligence

IQ Solutions, Inc.

11300 Rockville Pike, Suite 901

Rockville, MD 20852

Phone: (240) 221-4353

Fax: (240) 984-1473

Email: [email protected]

G. Project Statistician and Data Collection Entity

• SAMHSA Bullying Prevention App: User Feedback Survey: The contractor will collect and compile data using a web-based survey. The use of compiled and analyzed results is at the sole discretion of OC.

• SAMHSA Suicide Prevention App: User Feedback Survey: The contractor will collect and compile data using a web-based survey. The use of compiled and analyzed results is at the sole discretion of OC.

• SAMHSA Contact Center Customer Satisfaction Survey: The contractor will collect and compile data using a web-based survey and an IVR-based survey. The use of compiled and analyzed results is at the sole discretion of OC.

List of Attachments

Attachment 1: SAMHSA Bullying Prevention App: User Feedback Survey

Attachment 2: SAMHSA Suicide Prevention App: User Feedback Survey

Attachment 3: SAMHSA Contact Center Customer Satisfaction Survey

7