Customer Satisfaction Survey Continuous Quality Improvement for a National
Technical Assistance Center in Children’s Behavioral Health
A. Product/Activity to be Assessed
The Substance Abuse and Mental Health Services Administration’s (SAMHSA), Center for
Mental Health Services’ (CMHS) Child Adolescent and Family Branch (Branch) promotes and ensures that the mental health needs of children and their families are addressed by a community-based system of care. A component of the Branch’s work involves a range of activities designed to provide comprehensive technical assistance (TA) to CMHS system of care and expansion grantees. The feedback from these grantees is vital to the improvement of CMHS grantee services. For this reason, the Branch is requesting OMB approval for two customer satisfaction feedback surveys and one case study protocol designed to provide input for the Branch’s TA contractor the Technical Assistance Network for Children’s Behavioral Health (TA Network).
The TA Network is the coordinating entity for the Children’s Mental Health Initiative (CMHI) and is an alliance among organizations and individual experts in the systems of care (SOC) arena. TA Network partners include the National Federation of Families for Children’s Mental Health; the Family-Run Executive Directors Leadership Association (FREDLA); Youth MOVE National; the University of South Florida, Department of Child and Family Studies; Portland State University; the University of Washington, and the National Wraparound Initiative; Accountability Solutions, Inc.; the Center for Health Care Strategies, Inc.; Human Service Collaborative; and, the Center for Community Learning, Inc., along with an extensive consultant pool.
The TA Network has the primary objective of assisting state with “scaling up” within SOC grantees to support statewide implementation along with building local capacity. This aligns with SAMSHA’s “scaling up” of the CMHI Program as demonstrated through the SOC Expansion Planning Grants and SOC Expansion Implementation Cooperative Agreements. The continuous quality improvement (CQI) protocol for this project primarily supports SAMSHA’s Aim of Achieving Excellence in Operations with an emphasis on data-driven organizational structures and processes. The data gathered through the CQI satisfaction surveys will guide the TA Network’s internal improvement process for delivering TA to SOC grantees by applying data analysis and systematic reporting to inform the CQI efforts.
The CQI satisfaction surveys will assess the overall satisfaction and impact of the TA Network’s technical assistance that is provided to SOC grantees. The TA Network, housed within the University of Maryland, School of Social Work submitted the protocol described in this supporting statement to the University of Maryland (UM) Institutional Review Board (IRB). The UM IRB determined that this CQI satisfaction survey met the definition of non-human subjects research and the TA Network is able to proceed within the UM structure without IRB oversight (Appendix D). The CQI satisfaction survey will apply a systematic approach of infusing CQI into all TA Network processes from the outset. The CQI satisfaction survey consists of obtaining structured input from SOC grantees receiving TA from the TA Network as well as a small case study component to further the TA Network’s understanding of the grantees TA utilization experiences.
TA Network is requesting OMB approval for:
Annual Generalized Technical Assistance Survey (Appendix A) will be conducted on an annual basis to gain insight about the satisfaction of all of the TA provided by the TA Network. The survey will be administered in an online format that website users can access annually.
Quarterly Impact of Training and Technical Assistance Survey (IOTTA) (Appendix B) designed to assess grantee satisfaction and the impact of the TA provided by the TA Network. The survey will be administered in an online format that website users can access following their experience receiving TA.
Case Study Protocol/Semi-structured Focus Group Interviews (Appendix C) will be conducted to gain in-depth insight on the satisfaction and relevance/impact of TA received from the TA Network.
Annual Generalized Technical Assistance Survey:
Annual Generalized Technical Assistance Survey (Appendix A) is designed to access whether the TA Network provision of all TA meets consumer needs in terms of quality, utilization, satisfaction. The Annual Generalized Technical Assistance Survey (AGTAS) includes fill-in-the-blank items, some Likert scale items, as well as pre-populated drop-down options. Specifically, the AGTAS assesses:
Basic information about the participant (no identifying information)
Topic areas in which users accessed TA
Frequency accessing individualized TA
Perceived quality of individualized TA
Frequency accessing general TA
Perceived quality of general TA
Impact of the TA provider transition
Quarterly Impact of Training and Technical Assistance Survey:
The Quarterly Impact of Training and Technical Assistance (IOTTA) Tool (Appendix B) is designed to determine whether the TA Network TA interactions meet consumer needs in terms of accessibility, usability, quality, and satisfaction of TA. The IOTTA includes fill-in-the-blank items, some Likert scale items, as well as pre-populated drop-down options. Specifically, the IOTTA assesses:
Understanding/Mastery of TA topic before and after receipt of TA
Usefulness of TA
TA Provider Professionalism
TA Provider Credibility
TA Organization
Perceived Level of Impact on Recipients’ Work
Perceived Impact on the participant’s System of Care community
Case Study/Semi-Structured Focus Group Interview:
The Case Study/Semi-Structured Focus Group Interview (Appendix B) will be conducted to gain insight on the satisfaction and relevance/impact of TA received from the TA Network.
The Case Study Protocol/Semi-structured Focus Group Interview process will focus on the following themes (in the form of questions):
What was the decision-making process about whether or not to request TA assistance from the Network, and what were the rationales for decisions that were made?
What were the purposes of the TA and how well were they achieved?
What preparation was done for the TA that was provided, and how useful was the preparation?
How closely did the TA that was provided fit the values, goals and strategies of the recipients?
If multiple TA providers were used, how consistent were they in the information they provided and the methods they used?
How consistent was the TA with the values, goals, logic model, and theory of change of the TA Network?
What is the likely impact of the TA both in meeting a short-term need, and in enhancing the overall capacity of the recipient?
What lessons can be learned from the experience of providing TA to this recipient either about improving the TA process or strengthening system change efforts?
The case study protocol will be initially used on a pilot basis with two grantees, and after the pilot is completed and appropriate modifications are made, it will be used with nine grantees per year. The case studies will be randomly selected and initiated at the rate of one per month.
The findings from the interviews and from the reviews of the surveys will be summarized by main themes by two reviewers. Reviewers will select specific statements from the interviews that best address each of these themes/questions and will also rate, on a seven-point scale, the following:
Overall degree of fit of the TA with the recipients values, goals and strategies
Overall degree of consistency of the TA with the Network’s values, goals and theory of change
Overall quality of the TA experience and satisfaction with it, as reflected by the recipient
Overall quality of the TA experience, as judged by the reviewers,
Magnitude of positive change likely to occur as a result of the TA.
B. Brief Statement of Objectives
These surveys will assess participant satisfaction with the contractor’s technical assistance services and products. The data collected from the surveys will include the following components:
Annual Generalized Technical Assistance Survey
Frequency—responses will determine mean frequencies across the topics and TA modalities in which the participant participated.
Specific satisfaction – responses will determine mean satisfaction across multiple items, indicating the extent to which participants are satisfied with aspects of: 1) Specific topical TA, 2) Individualized TA, 3) General TA, and 4) the impact of the provider transition
The contractor will disseminate the survey to participants on the TA Network listserv who will receive an individual email through Qualtrics with a link to complete the survey.
The contractor will use the data gathered by the survey to continuously improve TA services. To analyze the data, the contractor will collect aggregate data and compute descriptive statistics (e.g., mean, median, and frequency, scale and summarize the open-ended comments.
Quartly Impact of Training and Technical Assistance (IOTTA) Tool
Specific satisfaction – responses will determine mean satisfaction across multiple items, indicating the extent to which participants are satisfied with aspects of: 1) Understanding/Mastering of TA topic before and after receipt of TA, 2) Usefulness of TA, 3) TA Provider Professionalism, 4) TA Provider Credibility, 5) TA Organization, 6) Perceived Level of Impact on Recipients’ Work, 7) Perceived Impact on Systems of Care community
Technical Assistance Reporting System (TARS) tracks and coordinates the provision of all TA provided to individuals, organization, localities and states. TARS runs a report to gather information for the topics of the TA interaction.
The contractor will use the data gathered by the survey to continuously improve TA services. To analyze the data, the contractor will collect aggregate data and compute descriptive statistics (e.g., mean, median, and frequency, scale and summarize the open-ended comments.
Case Study Protocol/Semi-structured Focus Group Interviews
Reviewer’s Ratings – reviewers will select specific statements from interviews from each of themes/questions. They will also rate on a 7-point scale the following: 1) Overall degree of fit of TA, 2) Overall degree of consistency of TA with the Network’s goals, 3) Overall quality of TA experience and satisfaction with it, 4) Overall quality of TA, as judged by reviewers, and 5) Magnitude of positive change.
Specific & Personal satisfaction– participants’ comments and recommendations will provide rich, contextualized information about the products.
The Branch and the contractor will use the data gathered by the survey to continuously improve TA products. To analyze the data, the contractor will provide a thematic analysis of qualitative data and compute reviewers’ ratings (e.g., mean, median, and frequency) for the Likert scale items.
C. Overview of Methods to Collect the Information
Data collection method.
Annual Generalized Technical Assistance Survey: The TA Network will disseminate the AGTAS to participants who are included in the TA Network Listserv. These individuals will first receive a pre-notification email from the TA Network notifying them that they will soon receive a survey from Qualtrics. Two days following this notification they will receive the survey link. They will receive reminder emails on two occasions: 1 week following the initial dissemination and then the 2nd week following the initial dissemination if they have not completed the survey. The survey will remain open for 1 month.
Following close of the survey, the TA Network will analyze the data and will generate a report that will be provided to the TA Network Advisory Board and TA Network Management Team to continually enhance and modify the ways in with TA is provided to SOC grantees. The data collected will be aggregated and there will not be any collection of identifying information from participants.
Quarterly Impact of Training and Technical Assistance (IOTTA): The TA Network will disseminate the IOTTA (Appendix B) at 3 month intervals to individuals working within SOC grantee communities who have received TA from the TA Network during each quarter. In most instances, participants will receive this survey one or two times during the course of one year. The IOTTA tool allows for the assessment of the perceived quality and positive impact of a range of different types of training, coaching, or TA activities provided as part of a workforce development effort.
The TA Network TA providers are required to enter contact notes on the TA that they provided into the online Technical Assistance Reporting System (TARS). TARS is designed to centralize, track, and coordinate the provision of all TA provided to individuals, organizations, localities, and states. TARS allows for TA providers to enter and track details on TA provision and centralize the sharing of these data across all TA providers, the SOC grantee communities and program partners. Each quarter, the TA Network staff will run a report from TARS that will include the name of the TA provider, the individual receiving TA and the topics of the TA interaction.
Using the TARs report, an individualized IOTTA survey will be generated and disseminated to participants who have received TA over the previous 3 month period. The survey will be administered online using Qualtrics Survey Research Suite software. Qualtrics software is protected and is accessible using a secure username and password that is only known to the researcher. Qualtrics software allows for automatic assignment of a unique identification number for each email address and this allows the potential respondent to be tracked over time and for reminder emails to be disseminated only to those who have not yet completed or finished their survey.
During the subsequent 3 months, the TA Network will analyze the data and will generate a report that will be provided to the TA Network Advisory Board and TA Network Management Team to continually enhance and modify the ways in with TA is provided to SOC grantees. The data collected will be aggregated and there will not be any collection of identifying information from participants.
Case Study Protocol/Semi-structured Focus Group Interviews: The contractor will interview individuals (grantees) over the phone. This interviewer will have information from TARS to provide context and background information. Interviewees/informants will include the Project Director, the TA coordinator and at least one other person centrally involved in receiving TA or implementing the grantees systems change strategy. The case study protocol uses common language to solicit feedback regarding grantee satisfaction with TA products.
This interview protocol will initially be used on a pilot basis with two SOC grantees. After the pilot is completed and appropriate modifications are made, it will be used with nine grantees per year.
For each of the grantees participating in the Case Study, three to five informants will be identified to be interviewed individually on the phone. The informants will include the Project Director, the TA coordinator, and at least one other person centrally involved in receiving TA or implementing the grantee’s system change strategy. Prior to the beginning of the interviews, the TA reports from TARS for the sample sites will be reviewed. This will provide the interviewer with important context and background information.
The two reviewers will discuss their ratings, and reach agreement. The narrative information gathered from the interviews and the ratings provided by the reviewers will be synthesized without identifying information and shared with the TA Network Advisory Board and the TA Network Management Team which will examine them particularly to identify any lessons to be learned. In addition, the data collected from the CQI efforts will be included in regular reports to SAMHSA Center for Mental Health Services, Child, Adolescent and Family Branch.
Method for identifying respondents.
Annual Generalized Technical Assistance Survey: All individuals from grantee communities who are include on the TA Network Listserv will be surveyed on an annual basis.
Quarterly Impact of Training and Technical Assistance (IOTTA) Tool: All individuals from grantee communities who access TA through the TA Network will be surveyed on a quarterly basis.
Case Study Protocol/Semi-structured Focus Group: The grantees in the sample will include three (3) from each of these categories: local implementation grant; planning grant for statewide expansion; or implementation grant for statewide implementation. Case studies will be initiated at the rate of one per month beginning in January of each year. Grant communities will be selected at random within each of these categories.
Proposed sample size and rationale.
Annual Generalized Technical Assistance Survey: The contractor approximations that approximately 2000 individuals’ maximum per year from grantees will receive this survey. This is the total number of individuals included on the TA Network listserv. The contractor will ask all of these individuals to complete this survey, to help improve the quality of TA that is delivered. Assuming that 20% of these individuals complete this survey, 200 individuals from grantee communities included in the TA Network listserv will complete this survey that is administered annually.
Quarterly Impact of Training and Technical Assistance (IOTTA) Tool: The contractor approximations that approximately 400 individuals’ maximum per year from grantees will receive this survey. The contractor will ask all of these individuals to complete this survey, to help obtain representative feedback of all community members. Assuming that 35% of these individuals complete this survey, 140 individuals from grantee communities that access the TA Network will complete this survey that is administered quarterly, after receiving TA.
Case Study Protocol/Semi-structured Focus Group: The contactor's assumes that approximately 40 individuals from 6 different grantee sites will be interviewed or participate in the case study. The grantees in this sample will include three from each of these categories: local implementation grant, planning grant for statewide expansion, or implementation grant for statewide implementation. Grantees will be randomly selected from within each of these categories. For each of the grantees participating in the case study, 3-5 informants will be identified and interviewed on the phone. The informants will include the Project Director, the TA coordinator and at least one other person centrally involved in receiving TA or implementing the grantee’s systems change strategy. Assuming that 70% of these individuals fully participates in this case study, approximately 28 grantees will complete this case study each year.
Planned frequency of information collection.
Annual Generalized Technical Assistance Survey: The contractor will disseminate the AGTAS one time annually.
Quarterly Impact of Training and Technical Assistance (IOTTA) Tool: The contractor will administer the IOTTA every 3 months (quarterly) after receiving grantee information from TARS report. Each quarter, the TA Network staff will run a report from TARS that will include the name of the provider, the individual receiving TA and the TA interaction. From the TARS report, an individualized IOTTA survey will be generated and disseminated to participants who received TA over the 3-month period encouraging their participation in the survey.
Case Study Protocol/Semi-structured Focus Group: Each year case studies will be initiated and a sample will be randomly identified to participate in the interview over the phone.
Time period over which the information will be collected.
Annual Generalized Technical Assistance Survey: This survey will remain open for approximately one month.
Quarterly Impact of Training and Technical Assistance (IOTTA) Tool: This survey will remain active for approximately two weeks.
Case Study Protocol/Semi-structured Focus Group: This data collection period will remain active for the first month of the year in January, in an attempt to capture all identified participants in each of the six different grantee sites.
Expected response rate and plan for follow-up, if any, of non-respondents.
Annual Generalized Technical Assistance Survey: Based on past experience collecting similar feedback, the contractor anticipates that approximately 20% of grantee community members who are on the TA Network listserv will complete the survey. The contractor will send a pre-notification email to grantees that informs them to expect an emailed Qualtrics survey link within two days. Qualtrics will auto-generate an email and survey link to participants. One week and then two weeks following initial dissemination, participants will receive a reminder email to complete the survey. This email will again include a survey link. No further follow-up activities are proposed.
Quarterly Impact of Training and Technical Assistance (IOTTA) Tool Based on past experience collecting similar feedback, the contractor anticipates that approximately 35% of grantee community members will complete the survey. The contractor will send to grantees an email with a Qualtrics link to the survey. Within one week of the survey due date, the contractor will send an email reminding potential respondents from the grantee community to complete the survey. This email will again include a survey link. No further follow-up activities are proposed.
Case Study Protocol/Semi-structured Focus Group: The contractor estimates a higher response rate of individually reaching out to participants, at approximately 70% completion of telephone interviews for the case studies. For individuals selected to participate, they will receive reminder emails to follow up with scheduling the phone interview up to 2 times over a one month period.
Expected ability to assess non-response bias using existing information.
Annual Generalized Technical Assistance Survey: The contractor does not anticipate any non-response bias based on existing information.
Quarterly Impact of Training and Technical Assistance (IOTTA) Tool: The contractor does not anticipate any non-response bias based on existing information.
Case Study Protocol/Semi-structured Focus Group: The contractor does not anticipate any non-response bias based on existing information.
Methods used to maintain customer privacy.
Annual Generalized Technical Assistance Survey: Responses will be private. The introductory text for the survey explains this for respondents. Respondents will reply to the surveys using a common survey link, so it will not be possible to identify respondents unless they self-identify in the open-ended comments. Although not anonymous, the link has a unique identification number for each email address and customer responses will be private, with no anticipation of duplicate responses. Qualtrics software is protected and accessible using a secure user name and password that is only known to the researcher.
Quarterly Impact of Training and Technical Assistance (IOTTA) Tool: Responses will be private. The introductory text for the survey explains this for respondents. Respondents will reply to the surveys using a common survey link, so it will not be possible to identify respondents unless they self-identify in the open-ended comments. Although not anonymous, the link has a unique identification number for each email address and customer responses will be private, with no anticipation of duplicate responses. Qualtrics software is protected and accessible using a secure user name and password that is only known to the researcher.
Case Study Protocol/Semi-structured Focus Group: Responses will be private. Respondents will be recorded by phone and transcripts will be transcribed. Private and identifiable information will be private. This protocol has been noted as Non-Human Research and approved by the IRB at the University of Maryland, Baltimore. Additionally, storage of information from interviews will remain on password-protected databases.
D. Annual Response Burden Estimate
The following table provides information about the annual response burden estimated for these consumer feedback surveys.
Respondent |
Number of Respondents (a) |
Number of Responses/ Respondent (b) |
Total Responses |
Hours/ Response in Minutes (c) |
Annual Burden Hours |
Hourly Wage (d) |
Annual Cost ($) |
Annual Generalized Technical Assistance Survey |
200 |
1 |
200 |
.17 |
33 |
21.50 |
710 |
Quarterly Impact of Training and Technical Assistance (IOTTA) Tool |
400 |
2 |
800 |
.17 |
136 |
21.50 |
2,924 |
Case Study Protocol/ Semi-structured Focus Group |
40 |
1 |
40 |
1.5 |
60 |
21.50 |
1,290 |
Total |
640 |
|
1040 |
|
229 |
|
4,924 |
(a) The number of respondents has been estimated using the anticipated number of participants included in the sample population for the Generalized TA Survey (annual) and each of the IOTTA data collection periods (quarterly). For the case study protocol, the number of respondents is estimated for a 12-month period (40 respondents annually).
(b) For the purposes of estimating annual cost, the contractor assumes that community members will complete the Annual Generalized TA Survey one time per year and each IOTTA survey, a maximum of four times per year.
(c) The average burden per response was estimated based on independent review of the
surveys by contractor staff.
(d) The mean hourly wage is estimated at $21.50 for the types of respondents based on a mean yearly salary of $44,710 (assessed through a review of 2013 average salary estimates of community and social service occupations from the US Department of Labor’s Bureau of Labor Statistics, at http://www.bls.gov/oes/current/oes_nat.htm).
E. Methods Used to Develop and Test the Questions
Annual Generalized Technical Assistance Survey: The AGTAS was developed in part by the prior contractor for this TA center and was enhanced by the TA Network. Questions include typical types of satisfaction questions based on the type of TA provided through the TA Network. The contactor circulated several draft versions that were critiqued by the contractor’s management team and other staff for appropriate content and clarity of questions.
Quarterly Impact of Training and Technical Assistance (IOTTA) Tool: The IOTTA was developed by TA Network Core Partner, Portland State University School of Social Work. The IOTTA has been assessed for validity and reliability. More information on the IOTTA can be found at http://www.nwi.pdx.edu/pdf/IOTTA-results.pdf. The contractor made modifications to the IOTTA to ensure that the questions pertained to the participants in this CQI satisfaction survey. The contactor circulated several draft versions that were critiqued by the contractor’s management team and other staff for appropriate content and clarity of questions.
Case Study Semi-Structured Interview: The contractor prepared the semi-structured interview protocol. The contractor circulated several draft versions that were critiqued by the contractor’s management team and other staff for appropriate content and clarity of questions.
F. Federal Project Officer and Contractor
Diane Sondheimer
Deputy Chief
Child, Adolescent, and Family Branch
Federal Center for Mental Health Services
1 Choke Cherry Road Room 6-1043
Rockville, Maryland 20857
240-276-1922 Phone
240-276-1930 FAX
Michelle Zabel, MSW
Director, TA Network
The Institute for Innovation and Implementation
University of Maryland, School of Social Work
525 West Redwood Street
Baltimore, MD 21201
Phone: 410.706.4732
Marlene Matarese, PhD
Principal Investigator/ Deputy Director, TA Network
The Institute for Innovation and Implementation
University of Maryland, School of Social Work
525 West Redwood Street
Baltimore, MD 21201
Phone: 410.706.4732
Email: [email protected]
G. Project Statistician and Data Collection Entity
Annual Generalized Technical Assistance Survey: The contractor will collect and compile data using Qualtrics. The use of compiled and analyzed results is at the sole discretion of the Branch.
Quarterly Impact of Training and Technical Assistance (IOTTA): The contractor will collect and compile data using Qualtrics. The use of compiled and analyzed results is at the sole discretion of the Branch.
Case Study Protocol/Semi-structured Focus Group Interviews: The contractor will collect and compile data using Nvivo or another qualitative online coding program.
List of Attachments
Attachment 1: Annual Generalized Technical Assistance Survey (Appendix A)
Attachment 2: Quarterly Impact of Training and Technical Assistance (IOTTA) Survey TA Network CQI Survey (Appendix B)
Attachment 3: Case Study Protocol/Semi-structured Interviews (Appendix C)
Attachment 4: University of Maryland, Baltimore Institutional Review Board Determination (Appendix D)
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | SAMHSA Customer Satisfaction Survey Proposal |
| Author | Jennifer.D.Dewey |
| File Modified | 0000-00-00 |
| File Created | 2021-01-27 |