SUPPORTING STATEMENT
Part B
Assessing the Impact of the National Implementation of TeamSTEPPS Master Training Program
OMB CONTROL NO. 0935-0170
Version January 30, 2015
Agency of Healthcare Research and Quality (AHRQ)
Table of Contents
B. Collections of Information Employing Statistical Methods 3
1. Respondent universe and sampling methods 3
2. Information Collection Procedures 4
The training participant questionnaire will be distributed to all individuals who have participated in the National Implementation of TeamSTEPPS Master Training program since its inception in 2007. Currently, there are approximately 2400 anticipated training participants who will be studied under this effort. No sampling strategy will be employed, as AHRQ seeks to obtain feedback from everyone who has participated in the program (i.e., a census survey). A 75% response rate or higher is anticipated based on the positive relationship established between AHRQ, DoD, the training participants, and their organizations.
Semi-structured interviews will be conducted with nine organizations which had teams that participated in the TeamSTEPPS Master Training program. A two-stage purposive selection process will be used: first, organizations will be selected and second, individuals within the organization will be selected. The first stage of the selection process will select organizations so as to increase diversity with respect to the following factors: the type of implementation strategy used (i.e., a targeted strategy for dosing concepts across an organization or a transformational strategy for training all members of the organization on all concepts at once), size of training teams, leadership of teams and team composition, years in which the organization participated in the training, and geographic location. By applying these selection criteria, we maximize the variety of experiences that will be used to define a set of lessons learned from the organizations selected and from their respective organizations. The second stage of the selection process will also use purposive sampling. Interviewees will be selected from qualified individuals serving in a variety of roles (i.e., facilitators or implementers) among training participants. We will interview up to nine individuals per each of the nine organizations selected, for a maximum of 81 individuals.
Interviewees will include individuals from organizations participating in the TeamSTEPPS Master Training program as well as individuals from other organizations affected by patient safety initiatives that may have resulted from involvement with the TeamSTEPPS Master Training program. These individuals will be identified on a case-by-case basis using the list of training participants, information from publicly available written information on patient safety activities in the areas surrounding the organization, additional input from AHRQ and DoD to define a central organization and key contact, and referrals from the key contact in each organization.
Some of the limitations of this study, which will also be noted in the final report, include the following:
Retrospective analysis of post-training experiences. This is not an empirical study; therefore, neither causality nor generalizability can be established;
Survey non-response, which can limit our understanding of the concerns and issues participants encountered when trying to apply TeamSTEPPS tools and concepts in their home organizations or organizations that they support;
Lack of access to contact information for non-participants, except on the rare occasion when our site visit contacts identify non-participants to include in the interviews during the site visits;
Inclusion of only nine out of 52 states and territories for the site visits and qualitative interviews due to budgetary constraints. As a result, we will not be able to generalize findings beyond these individual cases;
Limited number of interviews that can be conducted at each site visit due to time and budgetary constraints; and
Inability to interview participants who have changed jobs since TeamSTEPPS Master Training. Because up to three years have passed for some of the participants, they may no longer work for the organization they were employed by at the time of participation or may no longer work in the field of patient safety.
Despite the limitations in this study, the data we collect via the questionnaire and semi-structured interviews will provide valuable information that will help develop tools and training that effectively support organizations’ efforts to improve patient safety. By understanding the factors that facilitate or inhibit the use of tools or the spread of knowledge, AHRQ will better understand the needs of these organizations and will be better prepared to address their future concerns, issues, and needs for improving patient safety.
The training participant questionnaire will be administered to all individuals who have participated in the TeamSTEPPS Master Training program using the Web-based questionnaire included in Attachment C. The participant questionnaire will be sent to all training participants via email. All correspondence to training participants is presented in Attachment D. These correspondences contain a hyperlink to enable easy access to the on-line questionnaire.
The training participant questionnaire will be accessible to training participants 24 hours a day for a total of 45 days. Upon entrance into the questionnaire, respondents will view an introduction page that explains the questionnaire objectives and stresses the importance of participation. Respondents will be able to easily respond to the questionnaire items by clicking on pre-coded options for closed-ended items and typing in "boxes" for any open-ended items.
Completed responses to the questionnaires will be backed up daily onto IMPAQ’s dedicated data collection server and will also be printed and stored in a locked cabinet. Data responses will be checked visually by researchers and analysts on a regular basis to assure that data are entered appropriately into the database.
Following data collection, training participant questionnaire responses will be compiled and assessed formally for data quality to produce a finalized database for statistical analyses. Incomplete response data poses a substantial threat to confident interpretation and generalization of the study results. The general approach to handling incomplete response data is to salvage as much data as possible using multiple techniques for examining patterns of missing data. Given the scope of the training participant questionnaire, IMPAQ will review questionnaire items with a substantial proportion of omitted responses. The precise cutoff percentage is typically chosen once the distribution of missing data has been established. It will be determined whether responses are missing in a manner that relates to other observable values. If data are determined to be missing in a manner that affects the interpretability of the responses, descriptive statistics and point estimates of relations among variables may be adjusted to account for missing data. This may be accomplished by using multiple imputation and full information maximum likelihood estimation techniques.
Semi-structured interviews will be conducted in person (or by phone when this is not possible) at one or more of the organizations selected. An experienced HRET/IMPAQ project team interviewer will conduct all interviews after obtaining written informed consent. Each interview will last about 60 minutes. Interviews will be digitally recorded and transcribed deleting any use of the interviewee’s name. Interviews, transcripts, and any additional notes from the interviews will be stored in secured computer servers at IMPAQ.
To effectively bolster the response rate, two interventions will be used: (1) time-staggered notices via email of the opportunity to participate in the questionnaire and (2) one telephone communication per invited respondent, encouraging them to participate in the AHRQ Web-based questionnaire of training participants. If at any point, an invited respondent refuses to participate, they will not be contacted again through any means.
During the data collection period, invited respondents who have yet to respond to the training participant questionnaire will be contacted via email reminding them of the opportunity to participate and the importance of their feedback regarding the training program. The re-contact notice sent via email will provide the hyperlink to access the questionnaire, the estimated time (in minutes) it will take to respond, the impending deadline for submission of their responses, and additional information regarding privacy of responses and confidentiality of personal information. The re-contact notice is provided in Attachment D.
It is important to note that the most common way to ensure an acceptable response rate is to provide introductory information through advance notice from a known and respected source – in this case, AHRQ and/or members of the National Implementation Program contract team. For the purposes of this study, the National Implementation Program’s project team will distribute an e-mail to all invited respondents to provide advance notice of the study and the importance of participation (see Attachment D).
Procedures to test the training participant questionnaire were conducted during the instrument’s development. Those procedures included two strategies: (1) a series of two to three cognitive laboratory interviews with AHRQ staff members and (2) a field test of the training participant questionnaire with a sub-sample of fewer than nine respondents. The goal of the cognitive laboratory was to identify questionnaire items and procedures that are confusing or intrusive, as well as questions that elicit ambiguous answers. During the cognitive laboratory, a researcher trained in the technique asked the participant to “think aloud” as he or she answered each question. In so doing, the researcher examined the thought processes of the respondent as he or she heard, interpreted, and decided on an answer. The results of the cognitive laboratory were used to refine the questionnaire prior to field-testing. Similarly, the field testing was conducted to ensure there were no issues with the usability of the instrument.
The proposed evaluation efforts represent a continuation of the evaluation of the National Implementation of TeamSTEPPS program. The current evaluation work (OMB No. 0935-0170, Expires 12/31/15) seeks further renewal to continue to evaluate this important, ongoing work for AHRQ. Because the questionnaire has been tested and used, no further testing procedures are required.
IMPAQ International will serve as the primary consultants for statistical aspects of the design and analysis of the web-questionnaire data. Dr. David P. Baker, Executive Vice President, IMPAQ International, is the primary point of contact for statistical design and analyses. He can be reached at 443-259-5134 or [email protected].
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | SUPPORTING STATEMENT |
| Author | wcarroll |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |