Fetal Alcohol Spectrum Disorder (FASD) Center for Excellence
Customer Satisfaction Feedback Forms for Training and Technical Assistance
Product/Activity to be Assessed
Under the Government Performance and Results Act (GPRA), the Substance Abuse and Mental Health Services Administration (SAMHSA) Center for Substance Abuse Prevention (CSAP) is responsible for monitoring program performance across all of its budgetary lines. In compliance with these requirements, CSAP’s Fetal Alcohol Spectrum Disorder (FASD) Center for Excellence proposes to collect customer satisfaction feedback after all training and technical assistance events that are four or more hours in length. Customer satisfaction data will be used to maintain accountability and performance assessments of CSAP’s programs, and to make such information available to CSAP Project Officers for improved project oversight and to CSAP management for improved staff and grantee portfolio oversight.
The purpose of the SAMHSA FASD Center for Excellence is to prevent FASD and improve the quality of life for individuals and families affected by FASD. As a cornerstone of the services delivered by the Center for Excellence, targeted training and technical assistance are provided in order to significantly improve immediate, intermediate, and long-term outcomes in FASD prevention and treatment. The overall goals of FASD training and technical assistance are to:
(1) strengthen the performance and accountability of FASD prevention and treatment services, including reaching women at risk of giving birth to a child with an FASD through such promising practices as Project Choices and the Parent Child Assistance Program (PCAP);
(2) provide training and technical assistance to support effective FASD prevention and treatment programs, including providing up-to-date information about FASD to individuals across the United States who have been identified as FASD trainers; and
(3) assess the impact of the training and technical assistance services through systematic data collection and data analysis.
CSAP and the FASD Center for Excellence are interested in conducting feedback surveys for service providers, individuals with an FASD and their families, States working to develop and implement FASD State plans, and members of the public who participate in GPRA Training and Technical Assistance (TA) events. Feedback will be about the content and materials provided during training and technical assistance events and their utility to respondents after they return to performing work pertaining to FASD prevention and treatment.
The FASD Center for Excellence (CFE) has been established as a result of legislative mandates from the Children's Health Act of 2000 (P.L. 106-310) The proposed data collection will enable the SAMHSA FASD Center for Excellence to monitor the delivery and quality of the services provided. Section 501 (d)(4) of the Public Health Service Act requires that the Secretary of DHHS, acting through the Administrator shall “. . . assure that the Administration conduct and coordinate demonstration projects, evaluations, and service system assessments and the activities necessary to improve the availability and quality of treatment, prevention, and related services. The request to receive OMB clearance to assess customer satisfaction is submitted in response to that requirement.
B. Statement of Objectives
SAMHSA/CSAP and the FASD Center for Excellence are proposing ten customer satisfaction instruments that will be used to assess training events and technical assistance services. The proposed system is designed to collect information that can be used both to track the quantity and frequency of the different types of training and technical assistance services provided, and to assess customer satisfaction and changes in knowledge (e.g., ability to identify methods to prevent FASD) achieved in response to the services provided. This information will be used by SAMHSA/CSAP to both evaluate the effectiveness and assess performance and progress.
Different forms will be used to assess training, technical assistance, and informational meetings with the FASD field trainers. The forms for which approval is requested are:
Trainings
Participant feedback is obtained for all training events that are 4 or more hours in length.
FASD Event Pre-Test Form [to be paired with FASD Event Posttest Form] (Attachment A): The FASD Event Pretest Form is designed to provide necessary data about participants’ knowledge prior to an event in order to assess changes in these areas after participation in the event. In order to fulfill contractual and GPRA reporting requirements, demographic information is also collected. SAMHSA/CSAP estimates that the FASD Event Pretest Form will be administered at 100% of the training events, or to an estimated 4,800 people over the course of the contract.
FASD Event Post-Test Form [to be paired with FASD Event Pretest Form] (Attachment B): The FASD Event Posttest Form is designed to assess changes in participants’ knowledge after participation in the training event when compared to pretest results. SAMHSA/CSAP estimates that the FASD Event Posttest Form will be administered at 100% of the training events, or to an estimated 4,800 people over the course of the contract.
FASD Event Feedback Form (Attachment C): The FASD Event Feedback Form asks participants to assess both the overall event and specific components of the event (e.g., opportunity for questions/discussion, handouts/materials); rate the speaker(s); discuss the most and least helpful features of the event, along with suggestions for improving future trainings; and to rate the amount of new information/ideas received. The form also includes an assessment of the event’s success in meeting its stated objectives. SAMHSA/CSAP estimates that the FASD Event Feedback Form will be administered to an estimated 4,800 people over the course of the contract.
FASD Training Feedback Survey (Attachment D): In order to assess the intermediate and long-term impact of trainings provided by the SAMHSA FASD Center for Excellence, the FASD Training Feedback Survey will be administered both three and six months following clients’ participation in a training event. The form allows respondents to: assess the utility of the information/ideas/skills received in the event; report on whether they have been able to, or plan to, apply such information/ideas/skills; evaluate the extent to which the information/ideas/skills have improved their capacity to provide effective prevention and treatment services; and provide additional open-ended comments or suggestions on how such service could be improved in the future. An invitation to participate in the follow-up survey, including a URL link to a web-based version of the FASD Training Feedback Survey, will be distributed via e-mail. The survey will be uploaded to a secure, encrypted website that can be viewed from any web browser. Participants who prefer not to complete the survey online will have the option to provide their feedback through an interview instead. All participants also have the option to opt-out if they prefer not to participate in the follow-up survey. Non-respondents will receive two e-mail reminders. Although in the past approximately 50% of participants completed the FASD Training Feedback Survey at each of the two follow-up periods (three months and six months), SAMHSA/CSAP expects to obtain follow-up feedback from 80% of the total respondents, or 3840 respondents.
Meetings
As part of the services provided by the Center for Excellence, the Center provides up-to-date information about FASD to individuals across the United States who have been identified as FASD trainers.
Pre-Meeting Form [Field Trainers] (Attachment E): The Pre-Meeting Form is designed to assess participants’ knowledge of essential facts in the field of FASD prior to a meeting, for comparison with their level of knowledge in these areas after participation in the informational meeting. In order to fulfill contractual and GPRA reporting requirements, information pertaining to demographic background and professional affiliation is also collected. SAMHSA/CSAP estimates that approximately 100 people will complete the Pre-Meeting Form.
Post-Meeting Evaluation Form [Field Trainers] (Attachment F): The Post-Meeting Evaluation Form, designed to be paired with the Pre-Meeting Form, is intended to assess changes in field trainers’ knowledge after participation in an informational meeting. Results of the Post-Meeting Evaluation Form are compared with results of the Pre-Meeting Form in order to assess changes in knowledge immediately following an event. The Post-Meeting Evaluation Form is also designed to allow participants to assess overall satisfaction with the meeting, including specific aspects such as handouts/materials and opportunities for questions/discussion; rate their satisfaction with the speaker(s); describe any potential barriers to applying the information presented in the meeting; provide open-ended comments about the most and least helpful features of the meeting; and detail any suggestions for improvement. SAMHSA/CSAP estimates that approximately 100 people will complete the Post-Meeting Evaluation Form.
FASD Meeting Follow-up Feedback Survey [Field Trainers] (Attachment G): In order to assess the intermediate and long-term impact of field trainer meetings, follow-up will be conducted both six and twelve months after the informational meeting. The FASD Meeting Follow-up Feedback Survey asks participants to reflect on the extent to which the information/ideas/skills presented in the meeting have been implemented; any barriers to implementation; and the extent to which the information/ideas/skills presented in the meeting have improved field trainers’ capacity to provide effective prevention and treatment services. An invitation to participate in the follow-up survey, including a URL link to a web-based version of the FASD Meeting Follow-up Feedback Survey, will be distributed via e-mail. The survey will be uploaded to a secure, encrypted website that can be viewed from any web browser. Participants who prefer not to complete the survey online will have the option to provide their feedback through an interview instead. All participants also have the option to opt-out if they prefer not to participate in the follow-up survey. Participants who opt out will be considered active non-respondents and will not be contacted further. Participants who do not complete a feedback form but do not opt out will be considered passive non-respondents. Passive non-respondents will receive one e-mail reminder. SAMHSA/CSAP estimates that approximately 80 people will complete the FASD Meeting Follow-up Feedback Survey at each of the two follow-up periods (six months and twelve months) over the life of the contract.
Technical Assistance
Technical Assistance Feedback Form (Attachment H): The Technical Assistance Feedback Form is designed to elicit feedback from clients who requested technical assistance services from the FASD Center for Excellence. The Technical Assistance Feedback Form asks participants to assess their satisfaction with the technical assistance and explain any reasons for dissatisfaction; rate the utility of the technical assistance and the extent to which the technical assistance improved participants’ project capacity; provide examples of how the technical assistance may be used to improve project capacity in the future; and discuss any potential barriers to utilizing the technical assistance, as well as strategies to overcome these barriers. In order to fulfill contractual and GPRA reporting requirements, demographic information is also collected. SAMHSA/CSAP estimates that approximately 1,500 will complete the Technical Assistance Feedback Form over the course of the contract.
FASD Technical Assistance Follow-up Feedback Survey (Attachment I): In order to assess the intermediate and long-term impact of technical assistance, follow-up will be conducted both six and twelve months after the technical assistance event. The FASD Technical Assistance Follow-up Feedback Survey asks participants to reflect on the extent to which the information/ideas/skills presented in the technical assistance event have been implemented; any barriers to implementation; and the extent to which the information/ideas/skills presented have improved participants’ capacity to provide effective prevention and treatment services. An invitation to participate in the follow-up survey, including a URL link to a web-based version of the FASD Technical Assistance Follow-up Feedback Survey, will be distributed via e-mail. The survey will be uploaded to a secure, encrypted website that can be viewed from any web browser. Participants who prefer not to complete the survey online will have the option to provide their feedback through an interview instead. All participants also have the option to opt-out if they prefer not to participate in the follow-up survey. Non-respondents will receive one e-mail reminder. SAMHSA/CSAP estimates that approximately 1200 people will complete the FASD Technical Follow-up Feedback Survey at each of the two follow-up periods (six months and twelve months).
All
Event Requestor Form (Attachment J): The Event Requestor Form is designed to solicit feedback from the administrative contact in the requesting organization to assess the administration of all events, including timeliness of the Center’s response to the request and overall quality of the assistance received from the Center. The Event Requestor Form will be used to monitor the quality of service provision for all events: trainings, informational meetings, and technical assistance events. SAMHSA/CSAP estimates that approximately 400 people will complete the Event Requestor Form over the course of the contract.
SAMHSA/CSAP will use the collected data both to respond to GPRA reporting requirements and to further establish an ongoing monitoring system that ensures that the most effective and cost-effective training and technical assistance are available to meet the goals of SAMHSA/CSAP, SAMHSA and HHS.
C. Methodology
Data Collection Method
Form |
Methodology |
Trainings |
|
FASD Event Pre-Test Form |
On-site hard copy or electronically when the event is provided electronically (e.g., webinar) at beginning of event |
FASD Event Post-Test Form |
On-site hard copy or electronically when the event is provided electronically (e.g., webinar) at end of event |
FASD Event Feedback Form |
On-site hard copy or electronically when the event is provided electronically (e.g., webinar) at end of event |
FASD Training Feedback Survey (3- and 6- month follow-up) |
Electronically 3 and 6 months after an event, with two reminder emails sent to non-respondents. Respondents have the option to complete survey via telephone interview. |
Meetings (Field Trainers) |
|
Pre-Meeting Form (Field Trainers) |
On-site hard copy or electronically when the event is provided electronically (e.g., webinar) at beginning of event |
Post-Meeting Form (Field Trainers) |
On-site hard copy or electronically when the event is provided electronically (e.g., webinar) at end of event |
FASD Meeting Follow-up Feedback Survey (6- and 12-month follow-up) |
Electronically 6 and 12 months after an event, with two reminder emails sent to non-respondents. Respondents have the option to complete survey via telephone interview. |
Technical Assistance |
|
Technical Assistance Feedback Form |
On-site hard copy or electronically when the event is provided electronically (e.g., webinar) at end of event |
Technical Assistance Follow-up Feedback Survey (6- and 12-month follow-up) |
Electronically 6 and 12 months after an event, with two reminder emails sent to non-respondents. Respondents have the option to complete survey via telephone interview. |
All Events (Trainings, Meetings, TA) |
|
Event Requestor Form |
Electronically to event coordinator at end of event |
Respondents
Because the proposed forms will be used to assess individual customer satisfaction in a variety of training settings, all respondents will be asked to provide feedback. (There will be no sampling.) However, respondents will also be told that their participation in the survey is voluntary.
Sample Size and Rationale
As illustrated in the table in section D (Burden Table), the potential respondent universe is expected to reach approximately 5,400 per fiscal year. This includes all individuals who participate in an informational meeting, training, or technical assistance event and choose to provide customer satisfaction feedback afterwards. Estimates of sample size were derived based on a review of the number of events and total number of participants served annually by training and technical assistance events that were provided under the previous FASD contract with CSAP.
Timeline of Data Collection
The training and TA surveys will begin after completion of the first training and TA sessions after OMB approval is obtained. The surveys will continue until the contract ends. At this time, the FASD contract has been funded by SAMHSA/CSAP until September 2012.
Response Rate
Given the brevity of the forms and the intention to collect data at the beginning and/or end of events, the FASD Center expects to achieve a 90% response rate for the FASD Event Pre-Test Form, FASD Event Feedback Form, FASD Event Post-Test Form, Pre-Meeting Form for Field Trainers, Post-Meeting Form for Field Trainers, Technical Assistance Feedback Form, and Event Requestor Form. This expectation is based on the Center’s previous experience assessing customer satisfaction using similar data collection methods.
Participant follow-up can result in slightly lower response rates due to respondent attrition. Therefore, the FASD Center expects to achieve a response rate of approximately 80% for the FASD Training Feedback Survey, FASD Meeting Follow-up Feedback Survey, and Technical Assistance Follow-up Feedback Survey. In order to maximize response rates, participants will be able to select the survey method that best facilitates their participation in the follow-up survey. Respondents can choose to complete the survey online via the internet or can participate in a telephone interview. Respondents also have the option to opt out of the survey if they choose not to participate. In order to maximize response rates, all respondents will receive two reminder emails. Such methods should ensure that 80% response rates are achieved.
Confidentiality
All survey instruments are completed on a voluntary basis. In order to protect the privacy of information obtained no identifying information will be requested from respondents. Instead of a personal identifier (such as their name), participants list the following five characters when completing a customer satisfaction form, which are concatenated to create a unique identification code:
First letter in mother’s first name
First letter in mother’s maiden name
First letter in the city of your birth
First letter in the state of your birth
First letter in your first name
The Event Requestor Form is the only form that contains organizational level information that is completed by the administrative contact for the training or technical assistance event. While the Event Requestor Form does not request identifying information, the administrative contact is likely to be a known entity. All respondents are assured that their names and other identifying information will not be transmitted in the data files submitted to SAMHSA. Attendance sign-in sheets that contain individually identifying information are not transmitted with the other forms and are stored in a locked file cabinet. Electronic copies of the questionnaire are stored in a password-protected file folder. Data files provided by the SAMHSA FASD Center for Excellence to the CSAP DACCC will not contain personal identifiers (other than the unique identifier code described above which cannot be linked to individuals).
To further ensure protection of individual responses, all data will be reported at the aggregate level so that individual responses cannot be identified; no data will be reported at the individual participant level or reported in such a way that individuals could be identified due to unique personal or organizational characteristics.
D. Annual Response Burden Estimate
The estimated annual hour burden for the proposed data collection is provided in the following table. The total annual average hour burden is expected to be approximately 601, with an estimated annual average cost of $10,706.
Two rates were used in estimating the total. First, an hourly labor rate of $13.91 was used when estimating the average annual cost of collecting respondent data for the Event Requestor Form. According to the Bureau of Labor Statistics’ National Occupational Employment and Wage Estimates for 2007, this rate represents the median hourly wage rate for office and administrative support occupations, which are the expected respondent pool for this form. An hourly labor rate of $17.87 was used when estimating the average annual cost of collecting respondent data for all other forms. This rate is the median hourly wage rate for individuals with community and social services occupations, who are the expected respondent pool for nine of the ten forms.
Instrument/Activity |
Number of Respondents |
Number of Responses per Respondent |
Average Hours of Burden per Response |
Total Burden Hours per Collection |
Hourly Wage |
Total Hour Cost |
|
Trainings |
|||||||
FASD Event Pre-Test Form |
1200 |
1 |
0.083 |
100 |
17.87 |
$1,780 |
|
FASD Event Post-Test Form |
1200 |
1 |
0.067 |
80 |
17.87 |
$1,437 |
|
FASD Event Feedback Form |
1200 |
1 |
0.17 |
204 |
17.87 |
$3,645 |
|
FASD Training Feedback Survey (3- and 6- month follow-up) |
960 |
2 |
0.067 |
129 |
17.87 |
$2,299 |
|
Meetings (Field Trainers) |
|
|
|
|
|
|
|
Pre-Meeting Form (Field Trainers) |
25 |
1 |
0.12 |
3 |
17.87 |
$54 |
|
Post-Meeting Form (Field Trainers) |
25 |
1 |
0.17 |
4 |
17.87 |
$76 |
|
FASD Meeting Follow Up Feedback Survey (6- and 12-month follow-up) |
20 |
2 |
0.067 |
3 |
17.87 |
$48 |
|
Technical Assistance |
|||||||
Technical Assistance Feedback Form |
375 |
1 |
0.083 |
31 |
17.87 |
$556 |
|
Technical Assistance Follow Up Feedback Survey (6- and 12-month follow-up) |
300 |
2 |
0.067 |
40 |
17.87 |
$718 |
|
All Events (Trainings, Meetings, TA) |
|||||||
Event Requestor Form |
100 |
1 |
0.067 |
7 |
13.91 |
$93 |
|
TOTAL |
5,405 |
|
|
601 |
|
$10,706 |
|
E. Methods Used to Develop and Test the Survey Questions
The proposed questionnaires and data collection procedures are based on the assessment forms and methods that have been used successfully by the CAPTs to collect customer satisfaction data. As a result, all measures are well tested and have been proven to work well for this type of assessment activity. No further pre-testing has been conducted.
F. Federal Project Officers
Patricia B. Getty, Ph.D.
Acting Director
Division of Systems Development
Center for Substance Abuse Prevention
Substance Abuse & Mental Health Services Admin.
One Choke Cherry Road, #4-1027
Rockville, MD 20857
(w) (240) 276-2577
(f) (240) 276-2580
Josefine Haynes-Battle, MSN, BSN, RN
LCDR, USPHS, Associate Recruiter
Public Health Analyst
Center for Substance Abuse Prevention
Division of Systems Development
1 Choke Cherry Road, Room 4-1026
Rockville, MD, 20857
(w) (240) 276-2563
(f) (240) 276-2410
List of Attachments
Attachment Description
A FASD Event Pre-Test Form
B FASD Event Post-Test Form
C FASD Event Feedback Form
D FASD Training Feedback Survey
E Pre-Meeting Form (Field Trainers)
G FASD Meeting Follow Up Feedback Survey
H Technical Assistance Feedback Form
I Technical Assistance Follow Up Feedback Survey
J Event Requestor Form
| File Type | application/msword |
| File Title | Fetal Alcohol Spectrum Disorder (FASD) Center for Excellence |
| Author | Vinitha Meyyur |
| Last Modified By | Vinitha Meyyur |
| File Modified | 2009-02-18 |
| File Created | 2009-02-18 |