Form User Feedback Form User Feedback Form User Feedback Form

OMB Customer Satisfaction Survey Clearance Request

For Suicide Prevention Assessment and Resource Kits (SPARK)

A. Product/Activity to be assessed

The purpose of the assessment is to evaluate the extent of consumer satisfaction with suicide prevention toolkits that are being developed for service providers of high risk populations, including senior living communities for older adults and school-based settings for youth. The development of the toolkits is being funded by the federal Substance Abuse and Mental Health Services Administration (SAMHSA), and they are being developed by the National Association of State Mental Health Program Directors (NASMHPD) and its subcontractors, the Education Development Center, Inc. (EDC), and McFarland and Associates. An additional subcontractor, the NASMHPD Research Institute, Inc. is responsible for the design and implementation of the dissemination, tracking and evaluation components of the project.

This application for OMB approval is specifically to conduct a survey for assessing consumer satisfaction with the first toolkit that is being developed for senior living communities that serve older adults during Fiscal Year 2008-2009. The toolkit will contain research-based guidelines and resources to help administrators, staff, and residents in senior living communities promote mental health and prevent suicide of the residents served. The survey will be conducted during Fiscal Year 2009-2010.

B. Objectives of proposed customer satisfaction effort

The objective of the proposed customer satisfaction effort is to assess the extent to which the toolkit materials provide helpful resources to enhance service providers’ capacity to respond to the needs of individuals for whom suicide could be a risk at three prevention levels: universal (i.e., whole population), selected (i.e., at risk), and indicated (i.e., response to crises and suicidal behaviors) levels. The results of the survey will be used by SAMHSA to enhance current and future efforts to disseminate suicide prevention information to the field. The survey will capture information about:

• The organization in which the toolkit is being used

• The target population served

• Respondent’s position in organization

• How the organization learned about the toolkit

• Extent to which specific sets of materials in the toolkit have been used in the setting

• Overall impressions of the format, organization, and content of the toolkit

• Perceived utility of the toolkit in building capacity of the organization to achieve suicide prevention goals

• Overall quality of the toolkit

• Organizational needs for more information or technical assistance

• Open comments

This application contains the survey form titled, User Feedback Survey - Promoting Mental Health and Preventing Suicide: A Toolkit for Senior Living Communities¹, which will be administered in a web-based format; as well as the correspondence that will be sent to respondents to invite them to participate in the survey and reminders.

C. Overview of data collection methodology

Identification of respondents - SAMHSA’s Health Information Network (SHIN) is the primary vehicle SAMHSA utilizes to disseminate and track products either developed by SAMHSA or by other entities under contract to SAMHSA. Through close coordination of our activities with SHIN we propose to invite customers that have requested the toolkit from SHIN to participate in the customer satisfaction survey. Special procedures will be established with SHIN to accomplish this, such as adding a step to the ordering process that informs the recipient that NRI will be conducting a customer satisfaction survey of the SPARK they are ordering and requests their permission for the evaluator to contact them to follow up after a specified period of time. If possible, this contact information should include name, email address, mailing address, telephone number, and type of senior living center for which they are requesting the toolkit. Therefore the total sample will be composed of the universe of recipients of each SPARK that give permission for follow-up contact by NRI.

Data collection - Data will be collected through a web-based survey. Provider organizations and individual professionals that ordered the SPARKS and provided permission for follow-up contact will be contacted through email (if email address is provided) or a hard copy letter (if mailing address is provided) to invite them to participate in the survey. The email or post card will briefly describe the purpose of the evaluation and will provide instructions for linking to the survey via a web address, as well as a unique password to login to the survey. Respondents will be tracked and non-respondents will be sent two rounds of reminder invitations. Non-respondents will then be contacted by telephone for a final invitation.

Recipients (i.e., potential survey respondents) will be contacted 3 months after they have ordered the SPARK. This will provide users a minimum amount of time to receive, review, and begin using the SPARK, and thus have sufficient experience to provide input on the satisfaction, use, utility, and potential impact of the SPARK.

Proposed sample size and rationale - The total possible sample size was estimated in two steps. The first consideration is the number of provider agencies that will order the toolkit. Although there are many associations targeted for dissemination activities, two in particular are considered to be the ones in which most of the senior living centers will have membership. These are the Assisted Living Federation of America (www.alfa.org) and the American Association of Homes and Services for the Aging (www.aahsa.org). Based on information obtained on their websites, it was estimated that together with state and local affiliates, these two associations have approximately 12,900 member agencies. If there is 25% overlap in the membership of these two associations (based on other evaluations conducted by NRI using professional associations to recruit respondent agencies), the possible number of agencies ordering the toolkit would be reduced to 9,675. However, we know that not all agencies will respond right away to the dissemination of information about the toolkit. Data collected by the NRI on uptake of various evidence-based practices (EBP) by State Mental Health Agencies (and their respective regional and local provider agencies) showed that in 2003, shortly after the first generation of EBP mental health toolkits were developed by SAMHSA, 36% of states reported implementing assertive community treatment at some level in their states. Using this 36% estimated adoption rate during the first year of toolkit release, the estimated number of agencies ordering the toolkit would then be reduced to approximately 3,500. From this estimated universe of agencies ordering the toolkit, the first 20% (N = 700) that order the toolkit will be invited to participate in the consumer feedback survey. Therefore for the purpose of estimating burden, we will use 700 as the possible total sample size.

Expected response rate and plan for follow-up, if any, of non-respondents – Response rates to web-based surveys have been reported to range between 20% and 80% (Cook, Heath, & Thompson, 2000; Dillman, 2000). Higher response rates are associated with multi-modal, repeated, and more personalized contacts (Cook, Heath, & Thompson, 2000; Dillman, 2000; Dillman et al., 2008). Cook and colleagues’ meta-analysis of web-based surveys reported a mean response rate of 39.6% (SD=19.6%) across 68 surveys. We anticipate a much higher response rate of 65% to 75% because we are only sending the survey to people who have already agreed to be contacted for follow-up surveys. Methods to be used to maximize response rates for this web-based survey will include a 4-stage approach including a formal individualized invitation, two follow-up reminders, and one telephone reminder. In addition, in the first email invitation respondents will be informed that they also have the option of responding via hard copy or telephone.

Expected ability to assess non-response bias using existing information – In the event that the response rate is lower than expected, non-response bias can be assessed by comparing respondents and non-respondents on data collected during the toolkit ordering phase such as locale, type of senior living center, and date materials were ordered.

Methods used to protect the identity of respondents – Respondents will be assigned unique passwords to access the web-based survey. This password will serve as their unique research ID which will be used in the research data base. A master list which links the research ID and the respondents’ contact information will be maintained in a locked file cabinet by the NRI Director of Special Studies who will be the Project Statistician.

Analyses - Quantitative data will be entered and analyzed in SPSS. Descriptive quantitative analyses will present the total number of respondents; the response rate; and means and standard deviations of specific items. Statistical analyses (independent t tests) will be conducted to examine differences in responses among different organizational types. Qualitative data will be analyzed using qualitative techniques to identify themes and tabulate responses in major theme areas.

D. Annual response burden estimate

Survey	Number of Respondents (a)	Responses per Respondent	Hours per Respondent (b)	Annual Burden Hours	Hourly wage (c)	Total Hourly Cost
Web-based	700	1	.25	175	$19.23	$3,365.25

(a) The maximum number of respondents has been determined based on sampling the estimated number of agencies that will order the toolkit and the estimated rate of adoption during the first year of toolkit release (See description above in Section C. “Overview of data collection methodology / Proposed sample size and rationale”)

(b) The average time to complete the survey is estimated to be 15 minutes or .25 hours.

(c) The estimated hourly wage was based on similar customer satisfaction surveys conducted with service organizations participating in SAMHSA-funded suicide prevention initiatives and is estimated at $19.23 for this type of respondent that works in a social service agency and earns an annual salary of $40,000.

E. Methods used to develop and test the questions

The survey was developed specifically to assess Promoting Mental Health and Preventing Suicide: A Toolkit for Senior Living Communities. Standard methods related to attitude scale development (DeVellis, 2003) were used to develop the questions. The questionnaire will be piloted prior to the survey administration and revised as needed.

F. Federal Project Officer

Tarsha M. Wilson, Ph.D., LGSW

Lieutenant Commander, USPHS

SAMHSA/CMHS/Division of Prevention Traumatic Stress and Special Programs/Suicide Prevention Branch

1 Choke Cherry, Rm. 6-1138, Rockville, MD 20857 (postal mail) or 20850 (FedEx/UPS/DHL)

240-276-1748

240-276-1890

[email protected]

 

G. Project statistician and data collection entity

Jeanne C. Rivard, Ph.D.

Director of Special Studies

NASMHPD Research Institute, Inc.

66 Canal Center Plaza, Ste. 302

Alexandria, VA 22314

Phone: 703-682-9468

Fax: 703-548-9517

Email: [email protected]

References

Cook, C., Heath, F., Thompson, R.L. (2000). A meta-analysis of response rates in web-or internet-based surveys. Educational and Psychological Measurement, 60, 821-836.

DeVellis, R.F. (2003). Scale development: Theory and applications. Second edition. Thousand Oaks, CA: Sage Publications.

Dillman, D.A. (2000). Mail and internet surveys: The tailored design method. NY: John Wiley & Sons.

Dillman, D. A., Phelps, G., Tortora, R., Swift, K., Kohrell, J., Berck, J. & Messer, B.L. (On-going 2008). Response rate and measurement differences in mixed mode surveys using mail, telephone, interactive voice response (IVR) and the internet [Online]. Available at: http://www.sesrc.wsu.edu/dillman/papers.htm.

NASMHPD Research Institute (2007). NRI Task 25 Report: URS and NOMS. Unpublished report. Alexandria, VA: Author.

List of Attachments

1. User Feedback Survey – Promoting Mental Health and Preventing Suicide: A Toolkit for Senior Living Communities

2. Email Invitation to Participate in the customer satisfaction survey of Promoting Mental Health and Preventing Suicide: A Toolkit for Senior Living Communities

3. Email Reminder #1

4. Email Reminder #2

5. Telephone Reminder #3

1 This title has not been approved by SAMHSA and is used only as a working title in this document until an approved title is determined.