SUPPORTING STATEMENT
Part A
“Evaluation of the Implementation of TeamSTEPPS in Primary Care Settings (ITS-PC).”
Version: May 09, 2014
Agency of Healthcare Research and Quality (AHRQ)
Table of Contents
1. Circumstances That Make the Collection of Information Necessary 3
2. Purpose and Use of Information 5
3. Use of Improved Information Technology 5
4. Efforts to Identify Duplication 6
5. Involvement of Small Entities 6
6. Consequences if Information Collected Less Frequently 6
8. Federal Register Notice and Outside Consultations 7
9. Payments/Gifts to Respondents 7
10. Assurance of Confidentiality 7
11. Questions of a Sensitive Nature 7
12. Estimates of Annualized Burden Hours and Costs 8
13. Estimates of Annualized Respondent Capital and Maintenance Costs 8
14. Estimates of Annualized Cost to the Government 9
16. Time Schedule, Publication and Analysis Plans 9
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, the Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
1. Research that develops and presents scientific evidence regarding all aspects of health care;
2. The synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and
3. Initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
As part of its effort to fulfill its mission goals, AHRQ, in collaboration with the Department of Defense’s (DoD) Tricare Management Activity (TMA), developed TeamSTEPPS® (aka, Team Strategies and Tools for Enhancing Performance and Patient Safety) to provide an evidence-based suite of tools and strategies for training teamwork-based patient safety to health care professionals. TeamSTEPPS includes multiple toolkits which are all tied to or are variants of the core curriculum. In addition to the core curriculum, TeamSTEPPS resources have been developed for primary care, rapid response systems, long-term care, and patients with limited English proficiency.
The main objective of the TeamSTEPPS program is to improve patient safety by training health care staff in various teamwork, communication, and patient safety concepts, tools, and techniques and ultimately helping to build national capacity for supporting teamwork-based patient safety efforts in health care organizations. Since 2007, AHRQ’s National Implementation Program has produced (and continues to produce) Master Trainers who have stimulated the use and adoption of TeamSTEPPS in health care delivery systems. These individuals were trained using the TeamSTEPPS core curriculum at regional training centers across the U.S. AHRQ has also provided technical assistance and consultation on implementing TeamSTEPPS and has developed various channels of learning (e.g., user networks, various educational venues) for continued support and the improvement of teamwork in health care. Since the inception of the National Implementation Program, AHRQ has trained more than 5,000 participants to serve as TeamSTEPPS Master Trainers.
Given the success of the National Implementation Program, AHRQ launched an effort to provide TeamSTEPPS training to primary care health professionals using the TeamSTEPPS in Primary Care version of the curriculum. Most of the participants in the current National Implementation Program’s training come from hospital settings, because the TeamSTEPPS core curriculum is most aligned with that context. Under this new initiative, primary care practice facilitators will be trained through a combination of in-person and online training. Upon completion of the course, these individuals will be Master Trainers who will (a) train the staff at primary care practices, and (b) implement or support the implementation of TeamSTEPPS tools and strategies in primary care practices.
As part of this initiative, AHRQ seeks to conduct an evaluation of the TeamSTEPPS in Primary Care training program. This evaluation seeks to understand the effectiveness of the TeamSTEPPS in Primary Care training and how trained practice facilitators implement TeamSTEPPS in primary care practices.
This research has the following goals:
1) Conduct a formative assessment of the TeamSTEPPS for Primary Care training program to determine what revisions and improvement should be made to the training and how it is delivered, and
2) Identify how trained participants use and implement the TeamSTEPPS tools and resources in primary care settings.
To achieve the goals of this project, AHRQ will train primary care practice facilitators using the TeamSTEPPS in Primary Care training curriculum. Primary care practice facilitators may voluntarily sign up for this free, AHRQ sponsored training. Training will be delivered through a combination of online and in-person instruction. Online training will cover the core TeamSTEPPS tools and strategies that can be implemented in primary care. In-person instruction will cover coaching, organizational change, and implementation science. Practice facilitators, who complete the training, will be surveyed six months post-training.
The TeamSTEPPS Primary Care Post-Training Survey is an online instrument that will be administered to all primary care practice facilitators who complete the TeamSTEPPS in Primary Care training. The survey will be administered six months after participants complete training. The TeamSTEPP Primary Care Post-Training Survey can be found in Attachment A.
This study is being conducted by AHRQ through its contractor, the Health Research and Education Trust (HRET) and HRET’s subcontractor, IMPAQ International, pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2).
This is a new data collection effort for the purpose of conducting an evaluation of TeamSTEPPS in Primary Care Training. The evaluation is formative in nature as AHRQ seeks information to improve the content and delivery of the training. Training will be provided through a combination of online and in-person instruction.
To conduct the evaluation, the TeamSTEPPS in Primary Care Post-Training Survey will be administered to all individuals who complete the TeamSTEPPS in Primary Care training six months after training. The survey assesses the degree to which participants felt prepared by the training and what they did to implement TeamSTEPPS in primary care practices. Specifically, participants will be asked about their reasons for participating in the program; the degree to which they feel the training prepared them to train others in and use TeamSTEPPS in the primary care setting; what tools they have implemented in primary care practices; and resulting changes they have observed in the delivery of care.
The TeamSTEPPS in Primary Care Post-Training Survey will be administered via the Web to participants.
In order to reduce respondent burden, the training participant questionnaire will be administered via the Web. Participant information acquired by HRET when participants enroll in the TeamSTEPPS for Primary Care training program will be used to develop the distribution lists. Each potential respondent will receive four e-mail contacts to encourage participation (i.e., an advance notice of the questionnaire, an initial invitation to complete the questionnaire, and two follow-up e-mails to remind respondents to complete the questionnaire).
Using an online system for data collection, rather than administering a paper-based questionnaire, makes completing and submitting the questionnaire less time-consuming for respondents. Any skip patterns included in the questionnaire (i.e., questions that are appropriate only for a subset of the respondents) will be automatically programmed into the Web-based form of the questionnaire, thereby eliminating any confusion during questionnaire completion. In addition, the contractors can also ensure that important items are not inadvertently skipped or ignored by setting software requirements to ensure proper completion of questionnaires based on specific respondent selections.
AHRQ is aware of two other evaluations of the TeamSTEPPS suite of training programs, each requiring an Office of Management and Budget (OMB) clearance package. Each project evaluates a different TeamSTEPPS training program and collects data from distinct populations of individuals who participate in training. These evaluations are described below.
Assessing the Impact of the National Implementation of TeamSTEPPS Master Training Program. This effort provides in-person training to participants using the TeamSTEPPS core curriculum. AHRQ proposes to repeat an earlier survey of program participants and has submitted its prior OMB clearance package #0935-0170 for renewal.
TeamSTEPPS 2.0 Online Master Training. This effort provides online training to participants using a newly developed online version of the TeamSTEPPS core curriculum (i.e., TeamSTEPPS 2.0). AHRQ proposes to conduct a formative evaluation of this Web-based version of TeamSTEPPS training. The survey used in the Assessing the Impact of the National Implementation of TeamSTEPPS Master Training Program effort has been slightly modified to align with the Web-based training program.
Besides the two evaluation efforts listed above, AHRQ intends to collect similar data on the TeamSTEPPS in Primary Care training program. This effort (as described in this submission package) will provide information on the activities in which participants of this training program engage post-training, as well as the corresponding outcomes. The survey approved for the Assessing the Impact of the National Implementation of TeamSTEPPS Master Training Program effort has been slightly modified to align with this program’s target participants and curriculum.
The information collected may involve small entities, as primary care practice facilitators may work for very large or small primary care practices. Individuals from small practices will be required to complete the same number of items as those from larger practices. For this study, only items that provide critical information for answering the study questions will be included.
This is a one-time collection to answer specific questions about the TeamSTEPPS in Primary Care Training program.
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
As required by 5 CFR 1320.8(d), notice was published in the Federal Register on May 21, 2014 for 60 days (see Attachment B). No comments were received
AHRQ consulted with the American Board of Internal Medicine to determine if any secondary data sources were available to answer the questions for this study. None were identified.
No remuneration of respondents is planned.
Individuals will be assured of the confidentiality of their replies under Section 944(c) of the Public Health Service Act. 42 U.S.C. 299c-3(c). That law requires that information collected for research conducted or supported by AHRQ that identifies individuals or establishments be used only for the purpose for which it was supplied.
Information that can directly identify the respondent, such as name and/or social security number will not be collected. Only basic demographic information will be collected for the purpose of describing the respondents. Participation will be voluntary and participants will be informed that their responses will be confidential. A statement of confidentiality will appear on the online survey and contain the following statement:
“The confidentiality of your responses are protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)]. Information that could identify you will not be disclosed unless you have consented to that disclosure.”
The HRET project team will collect all information, and response data will be stored on secure serves. Files will be password-protected. Only members of the project team will have access to the data. Only aggregated data will be reported to AHRQ.
Questionnaire items do not require respondents to provide information of a sensitive nature as defined by OMB and DHHS or to provide information such as social security numbers or Medicare/Medicaid numbers. The TeamSTEPPS in Primary Care Post-Training Survey includes an introduction that addresses aspects of informed consent such as a description of the research objectives, a discussion of the importance of their input and experiences, details concerning how the data will be used, and confidentiality of personal information. The introduction will be positioned at the beginning of the questionnaire. Continuation to complete the questionnaire will indicate the respondent’s consent.
Although written consent will not be required, all respondents will be informed about the nature of the study and that their participation is voluntary. There are no known consequences of participation, and all confidentiality procedures will be described.
Exhibit 1 shows the estimated annualized burden hours for the respondent’s time to participate in the study. The TeamSTEPPS in Primary Care Post-Training Survey will be completed by approximately 150 individuals. We estimate that each respondent will answer 20 items (i.e., number of responses per respondent) and responding to these 20 questions will require 20 minutes. The total annualized burden is estimated to be 50 hours.
Exhibit 2 shows the estimated annualized cost burden based on the respondents’ time to participate in the study. The total cost burden is estimated to be $4,348.
Exhibit 1. Estimated annualized burden hours
Form Name |
Number of respondents |
Number of responses per respondent |
Hours per response |
Total burden hours |
TeamSTEPPS in Primary Care Post-Training Survey |
150 |
1 |
20/60 |
50 |
Total |
150 |
NA |
NA |
50 |
Exhibit 2. Estimated annualized cost burden
Form Name |
Number of respondents |
Total burden hours |
Average hourly wage rate* |
Total cost burden |
TeamSTEPPS Primary Care Post-training Survey |
150 |
50 |
$86.95a |
$4,348 |
Total |
150 |
50 |
$86.95 |
$4,348 |
* National Compensation Survey: Occupational wages in the United States May 2012, “U.S. Department of Labor, Bureau of Labor Statistics.”
There are no direct costs to respondents other than their time to participate in the study.
Exhibit 3 presents that estimated total and annualized cost for the project.
Exhibit 3. Estimated Annualized Cost
Cost Component |
Annualized Cost |
Project Development |
20,594 |
Data Collection Activities |
2,877 |
Data Processing and Analysis |
7,147 |
Publication of Results |
3,577 |
Project Management |
7,356 |
Overhead |
20,772 |
Total |
62,323 |
Exhibit 4: Annual Cost to AHRQ
Tasks/Personnel |
Staff Count |
|
Annual Salary |
% of Time |
Cost |
Management Support: GS-14, Step 6 average |
1 |
|
$123,660 |
10% |
$12,366 |
Grand Total |
|
|
|
|
$12,366 |
This is a new collection of information.
The time schedule for the data collection, data analysis, and final report preparation is presented in Exhibit 4.
Exhibit 5. Timeframe for data collection, analysis, and preparation of final report
Data Collection and Analysis |
Timeframes |
Administer training participant questionnaire |
Immediately upon OMB approval |
Analyze data |
60 days from end of data collection |
Prepare final report |
90 days from end of data analysis |
IMPAQ will analyze the survey data to identify trends in usage of the TeamSTEPPS in Primary Care curriculum, as well as the perceived impact of the program on organizational outcomes. To that end, a three-phase analysis is proposed. That analysis includes (1) ensuring the quality of the data collected, (2) conducting descriptive analyses, and (3) conducting comparisons of specific Master Trainer types and cohorts. These phases of analyses are described below.
To ensure maximum integrity of the results, several data screening and checking procedures will be conducted (Tabachnick & Fidell, 1996). Specifically, data quality checks will be performed by searching for deviant response ranges, anomalous response patterns, excessive missing data, extreme outliers, and highly skewed or irregular distributions. From these analyses, faulty data or data of poor measurement quality will be flagged, corrected, and/or eliminated. For example, excessive missing data is an indicator of poor data quality. Respondents who fail to respond to more than 10 percent of the protocol questions on the Web survey will be identified; their pattern of responses will be reviewed more carefully to determine if, for example, the respondent’s data should be eliminated from the analyses. Obviously any strategies that result in the elimination of data would first be discussed with AHRQ representatives and then fully documented in the final report.
For the descriptive analyses, HRET will employ the following approach: (1) compute a number of descriptive statistics for each variable measured by the survey; (2) develop early warning data protocols (specific statistical analyses to indicate significant variability, low response rates, or error in the data); (3) conduct item analyses; and (4) conduct comprehensive group and subgroup analyses.
HRET will calculate frequency distributions, means, and standard deviations for each closed-ended item included in the survey and combinations of related items that focus on a particular variable or issue. In addition, these statistics will be calculated for each subgroup represented in the sample (e.g., year of training attendance) and conduct analyses to identify subgroup differences. Frequency distributions will show the percentage of people who responded to each response option for each item included in the protocol. Means and standard deviations will be used to examine the relative importance of different items and item combinations that measure specific issues associated with each survey. Finally, standard deviations will be used to examine the level of agreement among respondents regarding issues that are identified as important.
A few items included will be open-ended in nature as a means of following up on closed-ended items to obtain richer detail on unique activities being conducted post-training. Individual responses to the open-ended items will be recorded and compiled; these responses will be examined for any themes or patterns of interest. If appropriate, codes will be defined based on the themes identified and the open-ended responses would be coded into closed-ended categories, which would then be tabulated.
The data gathered from the survey will allow us to determine differences between roles that Master Trainers may fulfill. For example, comparisons of each TeamSTEPPS tool and strategy will be conducted based on the ratings of usefulness by averaging the usefulness ratings of each tool across all participants and then conduct t-tests to assess the magnitude of any subgroup differences. In addition to conducting comparisons of tool usefulness, comparisons of tool usage and perceived impact will also be conducted, and variations in these characteristics will be analyzed by training participant type. To identify the magnitude of any differences by training participant type, t-tests will be conducted. For example, the analyses will identify how useful a specific tool is for direct users/implementers as opposed to those Master Trainers whose roles are primarily to support others in the implementation or training of TeamSTEPPS.
AHRQ does not seek this exemption.
Attachment A -- TeamSTEPPS in Primary Care Post-Training Survey
Attachment B -- Federal Register Notice
Attachment C -- Advanced Notice Email
Attachment D -- Importance of Participation Email
Attachment E -- Reminder Email
Attachment F -- Thank you Email
.
| File Type | application/msword |
| File Title | OMB Clearance Application |
| Author | hamlin-ben |
| Last Modified By | erwin.brown |
| File Modified | 2014-08-18 |
| File Created | 2014-08-18 |