DTAC Event OMB SS-A 1-8-2014

SAMHSA Disaster Technical Assistance Center (DTAC)

Training, Webinar, Podcast, and Mobile Application Feedback Forms

Supporting Statement

A. Justification

A1. Circumstances of Data Collection

The Center for Mental Health Services (CMHS) within the Substance Abuse and Mental Health Services Administration (SAMHSA) is requesting clearance from the Office of Management and Budget (OMB) for data collection activities associated with the SAMHSA DTAC Training, Webinar, Podcast, and Mobile Application Feedback. The data collection will include the following five instruments:

• Training Feedback Form (Attachment A1)

• Webinar/Podcast Feedback Form (Attachment A2)

• Mobile Application Feedback Form (Attachment A3)

• Training Feedback Follow-Up Interviews (Attachment B1)

• Webinar Feedback Follow-Up Interviews (Attachment B2)

Established by SAMHSA, the Disaster Technical Assistance Center (DTAC) supports SAMHSA’s efforts to prepare states, territories, tribes, and local entities to deliver an effective mental health and substance abuse (behavioral health) response to disaster. SAMHSA provides disaster technical assistance, training, and consultation to help disaster behavioral health (DBH) professionals plan for and effectively respond to mental health and substance abuse needs following a disaster. SAMHSA supports all-hazards disaster behavioral health preparedness, response, and recovery through informed, targeted training and technical assistance to reduce the effects disasters can bring to communities, families, and individuals.

SAMHSA DTAC provides training, webinars, and podcasts in response to, and in preparation for, behavioral health (mental health and substance abuse) needs associated with catastrophic events and emergencies, such as natural disasters, human caused disasters, and technological disasters. These trainings, webinars, and podcasts (TWPs) cover various topics of interest to the disaster behavioral health fields, such as responding to specific types of disasters, working with specific groups of the population to prepare for or respond to disasters, and general disaster behavioral health challenges. The trainings are conducted either in person, virtually, or online, and the webinars and podcasts are delivered online. Webinars are live, interactive events with an audience viewing it real time while the podcasts are recorded events with no audience. Podcasts include recorded versions of live webinars, as well as other recorded events and meet section 508 requirements, such as, closed captioning. TWPs all use visual (PowerPoint presentations) and audio. In addition, SAMHSA is developing new applications that will provide DBH-related information to users via their mobile devices.

SAMHSA DTAC will be responsible for administering the proposed data collection instruments and analyzing the data collected from them. SAMHSA DTAC will use data from the forms and interviews to inform and enhance current and future TWP activities in terms of topic areas and content-delivery method.

The overall goal of this data collection effort is to inform SAMHSA’s second and seventh strategic initiatives: Trauma and Justice, and Data, Outcomes, and Quality. The effort specifically addresses Goal 2.5: Reduce the impact of disasters on the behavioral health of individuals, families, and communities; and Goal 7.4: Improve the quality and accessibility of surveillance, outcome and performance, and feedback information for staff, stakeholders, funders, and policymakers.

SAMHSA is requesting OMB review and approval of five data collection instruments. Table 1 provides an overview of the name of the instrument, the attachment reference, and the data collection method.

Table 1. Data Collection Instruments

Instrument	Attachment	Data Collection Method
Training Feedback Form	Attachment A1	Web/In-person
Webinar/Podcast Feedback Form	Attachment A2	Web
Mobile Application Feedback Form	Attachment A3	Web
Training Feedback Follow-Up Interview Guide	Attachment B1	Telephone
Webinar Feedback Follow-Up Interview Guide	Attachment B2	Telephone

A2. Purpose and Use of the Information Collected

SAMHSA DTAC has provided training and technical assistance to states since 2002. In addition to offering informational materials available by telephone, email, or web, SAMHSA DTAC presents trainings (both in person and online), webinars, and podcasts. The TWPs vary in length and cover a variety of DBH topics. While SAMHSA DTAC conducts periodic customer satisfaction forms to examine general satisfaction with various areas of SAMHSA DTAC services, SAMHSA DTAC has not gathered comprehensive feedback on the TWPs it provides. Some efforts to gather feedback have been conducted, but a more comprehensive assessment of the content and implementation of the TWPs is necessary to adequately inform SAMHSA DTAC and subsequently enhance the delivery of these programs. Gathering this feedback is necessary to examine the usefulness and value of the TWPs offered, including how useful they are in educating and assisting those who view or attend them and the impact they have in supporting jurisdictional planning efforts. SAMHSA is also currently developing applications that will provide DBH information to users through mobile devices. Feedback Forms of the usefulness of this mobile application is also necessary in understanding the effectiveness of such an application and ways to enhance the application and add additional content to address additional needs in the DBH field. The proposed data collection effort will provide SAMHSA DTAC with this much needed feedback, to help enhance current TWPs, as well as the mobile application currently being developed, and to help create additional useful TWPs and mobile applications moving forward. A summary of the specific instruments to be used in gathering feedback is provided below.

The Training Feedback Form (Attachment A1), Webinar/Podcast Feedback Form (Attachment A2), and Mobile Application Feedback Form (Attachment A3) are similar instruments, with minor changes made to accommodate the unique features of each event or feature. The Mobile Application Feedback Form is somewhat shorter than the other two instruments due to the type of technology to be used. The Training Feedback Form and the Webinar/Podcast Feedback Form are designed to elicit feedback on aspects of the content, presentation style, and presentation mode, as well as the relevance of and satisfaction with the information presented. The Mobile Application Feedback Form is designed to elicit feedback on the usefulness of and satisfaction with the application. Information gathered from these forms will be used to enhance SAMHSA DTAC TWPs and further develop or enhance the SAMHSA mobile applications.

Training Feedback Form. The Training Feedback Form will be delivered by email or hard copy (dependent upon event type and delivery method—in person or online) to all training participants and assesses the following:

• Method of learning about the training

• Participants’ primary job role

• Relevance of the information

• Amount of new information learned through the training

• Ability to apply the information to participants’ work

• Satisfaction with specific aspects of the training

• Suggestions for improving the training

• Opinions on the length of the training

• Overall satisfaction with the training

• Willingness to recommend the training to colleagues

Webinar/Podcast Feedback Form. The Webinar/Podcast Feedback Form will be administered by email or pop-up screen at the end of each webinar or podcast, dependent upon the type of technology used. The form assesses the following:

• Method of learning about the webinar or podcast

• Participants’ primary job role

• Relevance of the information

• Amount of new information learned through the webinar or podcast

• Ability to apply the information to participants’ work

• Satisfaction with specific aspects of the webinar or podcast

• Suggestions for improving the webinar or podcast

• Opinions on the length of the webinar or podcast

• Overall satisfaction with the webinar or podcast

• Willingness to recommend the webinar or podcast to colleagues

Mobile Application Feedback Form. The Mobile Application Feedback Form will be administered as a pop-up form directly through the mobile device application. The form assesses the following:

• Method of learning about the application

• Usefulness of the application

• Overall satisfaction with the application

• Suggestions for improving the application

• Users’ primary job role

The Training Feedback Follow-Up Interviews (Attachment B1) and Webinar Feedback Follow-Up Interviews (Attachment B2) are similar instruments, with minor changes in language based on the type of event. Both interviews will be administered by telephone approximately one month following the training. The interview guides are designed to facilitate a discussion that will delve deeper into the participant’s experiences with the training or webinar and its relevance to their job. Information gathered during these interviews will be used to enhance existing TWPs and guide the development of future TWPs. The Training Feedback Follow-Up Interviews and Webinar Feedback Follow-Up Interviews will assess the following:

• Participant’s background and experiences

• Level to which the training or webinar met expectations

• Recall of information learned during the training or webinar

• Ability to apply information learned during the training or webinar to job tasks

• Suggestions for improving SAMHSA DTAC training events

• Suggestions for future training or webinar topics

A3. Use of Information Technology

SAMHSA DTAC has made every effort to limit the burden on individual respondents through the use of technology. The Training Feedback Form, the Webinar/Podcast Feedback Form, and the Mobile Application Feedback Form will be administered by web, while the Training Feedback Follow-Up Interviews and Webinar Feedback Follow-Up Interviews will be administered by telephone. For in-person trainings, the Training Feedback Form may be administered using paper-and-pencil to minimize respondent burden.

Web-based Data Collection and Management

For TWPs, the Training Feedback Form and the Webinar/Podcast Feedback Form will be administered via the web. The web forms will include simplified screens and intuitive navigational controls (e.g., previous and next page buttons, progress bar) designed to achieve greater accuracy in response entry and greater respondent usability. The Mobile Application Feedback Form will be administered as a pop-up form directly through the mobile application. The look and feel of the web and mobile application form instruments will be customized using SAMHSA DTAC logos and colors, as appropriate.

Data for the web and mobile application-based administrations will be electronically gathered through the Internet. The electronic data will be stored on our secure server in a password-protected folder. No personally identifiable information (PII) or sensitive information (SI) will be requested. All form sample lists will be maintained in password-protected folders. Only authorized staff will be given access to the files.

Telephone-based Data Collection and Management

The Training Feedback Follow-Up Interviews and Webinar Feedback Follow-Up Interviews will be conducted by telephone. Data collected by telephone will be recorded for note taking purposes, if participants permit the researchers to do so. Electronic recordings will be stored in a password-protected folder and will be destroyed once the interview notes are finalized. Additional data will be recorded in electronic documents, which will also be stored in a password-protected folder. Participants will be recruited by email or phone, depending on available contact information. Contact information will be maintained in password-protected folders. Only authorized staff will be given access to the files.

A4. Efforts to Identify Duplication

The information to be collected is not available elsewhere and will be collected only for the purposes of this program.

A5. Impact on Small Business

The information collected will not have an impact on small business entities.

A6. Consequences of Collecting the Data Less Frequently

The current request represents ongoing data collection that will be used by SAMHSA DTAC to assess the development and delivery of SAMHSA DTAC TWPs and the SAMHSA mobile application. TWPs are continuously being developed, and the mobile application will continually be modified as additional materials of interest are developed, a one-time data collection is not sufficient for SAMHSA to evaluate the TWPs and mobile applications provided.

A7. Consistency with the Guidelines of 5 CFR 1320.5(d)(2)

The data collection fully complies with 5 CFR 1320.5(d)(2).

A8. Consultation Outside the Agency

a. Federal Register Notice

SAMHSA published a 60-day notice in the Federal Register on September 3, 2013 (78FRN 54261), soliciting public comment on this data collection. SAMHSA received no comments on the planned data collection.

b. Consultation Outside the Agency

Consultation on the design, instrumentation, and statistical aspects of the feedback forms has occurred with individuals outside of SAMHSA. The ICF International consultants are listed below:

Amy E. Falcone

Senior Associate

ICF International

9300 Lee Highway

Fairfax, VA 22031

Tel. 703-934-3935

Michael A. Lodato

Manager

ICF International

9300 Lee Highway

Fairfax, VA 22031

Tel. 703-934-3794

Amy R. Mack

SAMHSA DTAC Project Director

ICF International

9300 Lee Highway

Fairfax, VA 22031

Tel. 202-294-1431

A9. Payment or Gifts to Respondents

No payments or gifts will be offered or provided to respondents.

A10. Assurances of Privacy

Before conducting any data collection, SAMHSA will obtain informed consent to participate in data collection from potential respondents. Web-based data collection will be implemented for the Training Feedback Form and the Webinar/Podcast Feedback Form. Descriptive information will be collected from respondents, but no identifying information will be entered or stored into the web-based data repository. Telephone-based data colelction will be implemented for the Training Feedback Follow-Up Interviews and the Webinar Feedback Follow-Up Interviews. All staff members involved in data analysis are required to undergo security awareness training provided by ICF International. Participant names will not be connected to form data or reported in the results. Any form data or reported results shared outside of SAMHSA will be aggregated at the national level. Specific procedures to protect the privacy of respondents are described below for each data collection activity.

Training Feedback Form. The Training Feedback Form may be administered in either web-based or paper-and-pencil modes, dependent on event type and delivery mode. For paper administrations, the forms for all participants will be collected together and provided to the form Feedback Form team in a sealed envelope. Participants who attend in-person trainings may be asked to provide their contact information, if they are willing to participate in a Training Feedback Follow-Up Interview. Data files that contain this information will be maintained in password-protected folders. Only authorized staff will be given access to the files, on an as-needed basis. For web-based data collection, each respondent will be sent a personalized link to the form; however, form sample lists and form responses will be stored in separate password-protected folders. Informed consent will be established through the first screen of the web form, with participants providing consent to participate prior to beginning the form.

Webinar/Podcast Feedback Form. The Webinar/Podcast Feedback Form will be administered by email or pop-up screen at the end of the webinar or podcast, dependent upon the type of technology used. For forms administered by email, each respondent will be sent a personalized link to the form; however, form sample lists and form responses will be stored in separate password-protected folders. For forms administered by pop-up screen at the end of the webinar, the form will automatically be displayed at the end of the webinar. In both cases, informed consent will be established through the first screen of the web form, with participants providing consent to participate prior to beginning the form.

Mobile Application Feedback Form. The Mobile Application Feedback Form will be administered as a pop-up form directly through the application. The form will periodically be displayed to users as they are accessing the application. Users will have the ability to complete the form the first time it is displayed, select to have the form displayed again at a later time, or select to not complete the form.

Training FeedbackFollow-Up Interviews and Webinar Feedback Follow-Up Interviews. The Training Feedback Follow-Up Interviews and Webinar Feedback Follow-Up Interviews will be telephone-based. Sample lists with participant contact information used to schedule the interviews will be maintained in password-protected folders. Participants will be asked to provide permission to record the interviews; recordings will only be collected if participants provide permission to do so. Recordings obtained during the telephone interviews that contain PII, SI, or both, will be maintained in password-protected folders. Only authorized staff will be given access to the files. Furthermore, access to the files containing PII and SI will be granted only on an as-needed basis and only to those with the necessary clearance to handle the data.

A11. Questions of a Sensitive Nature

No information of a sensitive nature is being collected.

A12. Estimates of Annualized Burdens and Costs

Table 2 shows the estimated burden associated with the five data collection activities. The Training Feedback Form, Webinar/Podcast Feedback Form, Training Feedback Follow-Up Interviews, and Webinar Feedback Follow-Up Interviews will be administered following each training or webinar event, and periodically as podcasts are accessed. The Mobile Application Feedback Form will be administered periodically as users download and use the application.

SAMHSA DTAC provides trainings and webinars to help with both DBH preparedness and response. It is estimated that SAMHSA DTAC will offer an average of one scheduled training and five webinars/podcasts each year.  Based on participation in previous trainings and webinars, an average of 450 participants are expected to attend or view each training or webinar/podcast. Individuals will be able to attend or view all TWP events; however, due to the variety of topics covered in the events and the various occupational, personal, and research interests of the participants, it is anticipated that each participant will view or attend an average of three webinar(s)/podcast(s) per year.  The Training Feedback Form and the Webinar/Podcast Feedback Form will be administered to all participants in each TWP.  The Training Feedback Follow-Up Interviews and the Webinar Feedback Follow-Up Interviews will be administered to up to 10 percent of the participants from each training or webinar, or to 5 participants, whichever is greater.  Only one interview of each type will be requested from each participant per year.  The Mobile Application Form will be administered once during each year in which the application is used on the mobile device.

Table 2 also shows the associated burden for the Training Feedback Form and the Webinar/Podcast Feedback Form the Training Feedback Follow-Up Interviews and the Webinar Feedback Follow-Up Interviews, and the Mobile Application Form.  All participation estimates are based on attendance of previous events. The hourly wage rates for the instruments were calculated by averaging the salary of 12 positions from a broad sample that included different job categories.  The broad sample was selected because TWP participants and mobile application users include individuals from a wide variety of job categories.  The salary data were taken from publicly available sources.  The response time burden estimates are based on pre-tests with ICF International Survey Research Center staff members.

Table 2. Annualized Estimate of Respondent Burden

Instrument	Number of Respondents	Number of Responses per Respondent	Total Number of Responses	Hours per Response per Respondent	Total Burden Hours		Hourly Wage Rate1		Total Cost
Training Feedback Form	1,500	1	1,500	0.25	375.0		$35		$13,125.00
Webinar/Podcast Feedback Form	1,125	2	2,250	0.25	562.5	$35		$19,687.50
Mobile Application Form	600	1	600	0.25	150.0		$35		$5,250.00
Training Feedback Form Follow-Up Interviews	150	1	150	0.50	75.0		$35		$2,625.00
Webinar Feedback Form Follow-Up Interviews	225	1	225	0.50	112.5		$35		$3,937.50
Annual Total	3,600		4,725		1,275				$44,625.00

¹Wage data sources: Bureau of Labor Statistics. National compensation survey. Retrieved from http://www.bls.gov/ncs/; O*NET OnLine. (2010). Occupations [Quick search for occupations matching ‘substance abuse’]. Retrieved from http://online.onetcenter.org/find/result?s=Substance+Abuse; Salary.com. Salary wizard: Community health director [Data report]. Retrieved from http://swz.salary.com/salarywizard/layouthtmls/swzl_compresult_national_HC07000465.html

A13. Estimates of Annualized Cost Burden to Respondents or Record Keepers

There are no startup or capital costs, nor are there maintenance costs to the respondents.

A14. Estimates of Annualized Cost to the Government

CMHS has planned and allocated resources for the management, processing, and use of the collected information in a manner that shall enhance its utility to agencies and the public. Table 3 shows the associated government costs for the SAMHSA DTAC Training, Webinar, and Podcast Feedback Form Project.

It is estimated that CMHS will allocate 0.25 of a full-time equivalent each year for government oversight of the data collection. Assuming an annual salary of $80,000, these government costs will be $20,000 per year. The estimated annual cost for form development and maintenance, data collection, analysis, and report writing is $50,000.

Table 3. Annualized Estimate of Government Costs

	Total Cost
Government Oversight	$20,000
Contract Costs for Form Development and Maintenance, Data Collection, Analysis, and Report Writing	$50,000
Annual Total	$70,000

Total annual costs, including respondent burden and government costs, are estimated at $114,625.

A15. Changes in Burden

This is a new data collection.

A16. Time Schedule, Publication, Analysis Plans

a. Time Schedule

All instruments will follow the same time schedule, which is summarized in Table 4.

Table 4. Time Schedule

Task	Date
Obtain OMB Approval	Winter 2013
Data Collection	Spring 2014 through Spring 2016
Data Analysis	Ongoing
Annual Reporting	Ongoing

b. Publication Plans

There are no current publication plans for this data collection effort.

c. Analysis Plans

Training Feedback Form, Webinar/Podcast Feedback Form, and Mobile Application Feedback Form. Basic exploratory and descriptive analyses, including frequencies (i.e., the number or proportion of participants who provided each response to the item) and cross-tabulations (i.e., the proportion of participants within specific subgroups who provided each response to the item), will be used to analyze the form data from each event or application. Descriptive analyses will be conducted for the questions that address the feedback on content and presentation style and mode; usefulness and relevance of the information presented; and satisfaction. The form contains open-ended items (i.e., free-text response items) regarding suggestions for improving the presentation or application, reasons for recommending or not recommending the TWP or application to others, and additional comments. Open-ended items will be analyzed for major themes. The themes will be summarized, and the team will analyze the frequency and consistency of each theme. The results of this analysis will be presented in the study report using easy-to-read tables, graphs, and charts with explanatory text as appropriate.

Training Feedback Interviews and Webinar Feedback Interviews. The interviews consist exclusively of open-ended items. These items will be analyzed for major themes. The themes will be summarized, and the team will analyze the frequency and consistency of each theme. Analysis results will be presented in the study report using text-based explanations and selected interview quotations as well as easy-to-read tables, graphs, and charts when applicable.

A17. Display of Expiration Date

All data collection instruments will display the expiration date of OMB approval.

A18. Exceptions to the Certification Statement

This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.