Justification (SS)

SAMHSA CSAP Prevention Fellowship Program

Technical Assistance Feedback and Satisfaction Surveys

A. Product/Activity to be Assessed

The Substance Abuse and Mental Health Services Administration’s (SAMHSA), Center for Substance Abuse Prevention (CSAP) Prevention Fellowship Program (PFP) is a professional development program which addresses the critical shortage of substance abuse prevention professionals nationwide by providing training, mentorship, and hands-on work experience to develop a workforce of Certified Prevention Specialists. The Program’s work is centered on the provision of technical assistance (TA) to selected Prevention Fellows in the form of training, guidance and assistance in the adoption of core competencies for prevention practice, and real-life opportunities to apply these competencies. The feedback of these fellows is vital to ensuring the quality of the training and services provided, to ensuring that the program is achieving its intended purposes, and to the continual improvement of program activities. For this reason, the program is requesting OMB approval for two customer satisfaction feedback surveys designed to provide input for the contractor administering the PFP:

• The Annual Fellowship Survey (Attachment 1) will assess fellow satisfaction with and quality of the training, mentorship, and field placement provided during the Fellowship. The survey will be administered at the end of each program year. It will include Likert scale items (strongly agree, agree, neutral, disagree, strongly disagree and n/a).

• The Alumni Follow-up Survey (Attachment 2) will assess the sustained program impact on employment opportunities, as well as alumni’s professional status and continued application of the knowledge and skills gained during the PFP. The survey will be administered annually. It will include multiple choice items, Likert scale items (strongly agree, agree, neutral, disagree, strongly disagree), and open-ended items.

B. Brief Statement of Objectives

These surveys will assess participant satisfaction with and benefits from the technical assistance services and products delivered through the PFP. The data collected from the surveys will include the following components:

• Annual Fellowship Survey

• Rate of agreement – responses will determine mean level of agreement with multiple statements about various aspects of the program design, trainings, and mentorship.

• Frequency of occurrence – responses will determine how frequently several scenarios occurred during the program year, providing insight into the quality of the field placement.

• Specific satisfaction – responses will indicate the extent to which participants are satisfied with the overall Fellowship experience.

• The contractor will use the data gathered from the survey to continuously improve the program design and TA services provided. To analyze the data, the contractor will compute descriptive statistics (e.g., mean, median, and frequency) for the Likert scale items.

• Alumni Follow-up Survey

  • Multiple choice – responses will categorize alums based upon several possible options, with regard to their reason for exiting the program, certification, professional field, and training others. These items will provide insight into program outcomes and sustainable program impacts.

  • Rate of agreement – responses will determine mean level of agreement with a statement regarding the program’s impact on increasing job opportunities.

  • Open ended – responses will provide rich, contextualized information.

  • The contractor will use the data gathered from the survey to continuously improve the program design and TA services provided. To analyze the data, the contractor will compute descriptive statistics (e.g., mean, median, and frequency) for the multiple choice and Likert scale items, and summarize the open-ended comments.

C. Overview of Methods to Collect the Information

Data collection method.

• Annual Fellowship Survey: The contractor will administer the survey electronically. Fellows will receive an email requesting their completion of the survey and an electronic link to it. The survey uses commonly asked questions to solicit feedback regarding Fellow satisfaction with and benefits from TA products and services.

• Alumni Follow-up Survey: The contractor will administer the survey electronically. Alumni will receive an email requesting their completion of the survey and an electronic link to it. The survey uses commonly asked questions to solicit feedback regarding the sustained program impact, as well as alumni’s professional status and continued application of the knowledge and skills gained during the PFP.

Method for identifying respondents.

• Annual Fellowship Survey: All individuals who participate in the program will be eligible and encouraged to complete the survey.

• Alumni Follow-up Survey: All individuals who have graduated or elected to resign from the program will be eligible and encouraged to complete the survey.

Proposed sample size and rationale.

• Annual Fellowship Survey: The contractor estimates that approximately 30 individuals will receive this survey (15 possible first-year fellows and 15 possible second-year fellows).

• Alumni Follow-up Survey: The contractor will distribute this survey to all alumni for whom an active email address is maintained. The contractor estimates that 53 of the total 81 alums will receive this survey. The contractor estimates that 34 alums will respond to this survey (a response rate of 65%). (The total number of alumni is expected to increase by 15 each year.

Planned frequency of information collection.

• Annual Fellowship Survey: The contractor will administer the Annual Fellowship Survey at the end of each program year.

• Alumni Follow-up Survey: The contractor will administer the Alumni Follow-up Survey annually.

Time period over which the information will be collected.

• Annual Fellowship Survey: This survey will remain active for approximately four weeks.

• Alumni Follow-up Survey: This survey will remain active for approximately four weeks.

Expected response rate and plan for follow-up, if any, of non-respondents.

• Annual Fellowship Survey: As current fellows are contractors, survey completion will be required and 100% response-rate is expected (15 respondents). Non-respondents will be resent the link to access the survey via email, and may be contacted via phone to as a reminder to complete the survey.

• Alumni Follow-up Survey: The contractor anticipates that approximately 65% of those who receive the survey invitation will complete the survey (49 on average respondents).  The contractor will send an email invitation with a link to the survey, to all alumni for whom the contractor maintains an active email address. Reminder emails with a link to the survey, will be distributed to non-respondents.  Follow-up calls may also be made to those individuals for whom an active phone number is maintained.

Expected ability to assess non-response bias using existing information.

• Annual Fellowship Survey: The contractor does not anticipate any non-response bias, as a 100% response rate is expected.

• Alumni Follow-up Survey: The contractor does not anticipate any non-response bias based on existing information.

Methods used to maintain customer privacy.

• Annual Fellowship Survey: Responses will be anonymous. The introductory text for the survey explains this, and requests that respondents not write their name on the survey. Although respondents will receive a unique survey link to allow the contractor to track non-respondents, responses will be aggregated and analyzed on a group level.

• Alumni Follow-up Survey: Responses will be anonymous. The introductory text for the survey explains this, and requests that respondents not write their name on the survey. Although respondents will receive a unique survey link to allow the contractor to track non-respondents, responses will be aggregated and analyzed on a group level.

D. Annual Response Burden Estimate

The following table provides information about the annual response burden estimated for these consumer feedback surveys.

Respondent	Number of Respondents (a)	Number of Responses/ Respondent (b)	Hours/ Response in Minutes (c)	Annual Burden Hours	Hourly Wage (d)	Annual Cost ($)
Annual Fellowship Survey	15	1	0.16	2.4	21.07	101
Alumni Follow-up Survey	49	1	0.05	2.5	21.07	36
Total	64			4.9		137

(a) The number of respondents has been estimated using the average total number of active fellows for the Annual Fellowship Survey.  For the Alumni Follow-up Survey, the number of respondents is estimated as 65% of those for whom active email addresses are maintained.

(b) For the purposes of estimating annual cost, the contractor assumes that fellows/alumns will complete each survey only once in a year.

(c) The average burden per response was estimated based on independent review of the

surveys by contractor staff.

(d) The mean hourly wage is estimated at $21.07 for the types of respondents based on a mean

yearly salary of $43,830 (assessed through a review of 2011 average salary estimates of community and social service occupations from the US Department of Labor’s Bureau of Labor Statistics, at http://www.bls.gov/oes/current/oes_nat.htm).

E. Methods Used to Develop and Test the Questions

• Annual Fellowship Survey: The contractor prepared the consumer feedback survey. The contactor circulated several draft versions that were critiqued by the contractor’s management team and other staff for appropriate content and clarity of questions.

• Alumni Follow-up Survey: The contractor prepared the consumer feedback survey. The contractor circulated several draft versions that were critiqued by the contractor’s management team and other staff for appropriate content and clarity of questions.

F. Federal Project Officer and Contractor

CDR Daniel Bailey,

Federal Project Officer

SAMHSA/CSAP

Division of Systems Development

1 Choke Cherry Road, Room 4-1004

Rockville, MD 20857

Phone: (240) 276-2439

Email: [email protected]

Sabrina Roundtree

Project Director

Synergy Enterprises, Inc.

8575 Georgia Avenue, Suite 1440

Silver Spring, MD 20910

Phone: (240) 485-1700 x3621

Fax: (240) 485-1709

Email: [email protected]

G. Project Statistician and Data Collection Entity

• Annual Fellowship Survey: The contractor will collect and compile data using a database management system embedded in the contractor’s website. The use of compiled and analyzed results is at the sole discretion of the Division of Systems Development.

• Alumni Follow-up Survey: The contractor will collect and compile data using a database management system embedded in the contractor’s website. The use of compiled and analyzed results is at the sole discretion of the Division of Systems Development.

List of Attachments

Attachment 1: Annual Fellowship Survey

Attachment 2: Alumni Follow-up Survey

5