Module 10

MEASUREMENT
(DRAFT)

SUBSECTIONS
•
•
•

Measurement in
TeamSTEPPS
A Model of Training
Evaluation
Available Measures
(Reactions, Learning,
Behavior, Results)

TIME: 50 minutes
VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTION

INSTRUCTOR OUTLINE: MEASUREMENT

Measurement
Slide

Instructor Note: This module presents information about
how to measure the impact of TeamSTEPPS and the tools that
are available to support evaluation. It is important that participants
learn how to assess the effects of their TeamSTEPPS
implementation so they can determine whether TeamSTEPPS is
working and producing the desired outcomes.
This module is primarily a lecture. It presents a well-established
approach for conducting training evaluation and describes the
available measures and their uses. There is an exercise at the
end of the module for individuals to apply what they have
learned.

MODULE
TIME:
50 minutes
MATERIALS:
• Evaluating
TeamSTEPPS
Worksheet

The Measurement module includes the content provided in the
outline below. Instructors should use the information below to plan
how the module will be taught within the time available.
Content
1.

Introduction

2.

Page #

Approx. Time

4

2 mins

Kirkpatrick’s Training
Evaluation Model

5- 6

8 mins

3.

Available Measures

7 - 19

30 mins

4.

Evaluating
TeamSTEPPS
Exercise

20 - 21

10 mins

Additional Resources: Below are additional resources you may
wish to use to obtain additional information:
• Kirkpatrick, D.L. (1967). Evaluation of training. In R.L. Craig
(Ed.). Training and development handbook: a guide to human
resource development. New York: McGraw-Hill.
• Salas, E., Almeida, S.A., Salisbury, M., et al. (2009). What are
the critical success factors for team training in health care? Jt
Comm J Qual Pat Saf 35, 398–405.
• Weaver, S.J., Salas, E., & King, H. (2011). Twelve best
practices for team training evaluation in health care. Jt Comm
J Qual Pat Saf 37, 341–9.

TeamSTEPPS for Long-Term Care | Measurement VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTION

D-10-3

Measurement

OBJECTIVES
SAY:
Upon completion of this module, participants will be able to:
• Describe the importance of measurement;
• Describe the Kirkpatrick model of training evaluation;

Slide

• Identify measures that can be used to assess the impact of
TeamSTEPPS;
• Describe the AHRQ Nursing Home Survey on Patient Safety Culture;
and
• Prepare a plan for determining if TeamSTEPPS worked.

D-10-4 VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTION TeamSTEPPS for Long-Term Care | Measurement

MEASUREMENT IN TEAMSTEPPS

Measurement

SAY:
Measuring the impact of your TeamSTEPPS implementation is
critical. Measurement enables you to answer questions such as
(1) Did TeamSTEPPS work?; (2) Did TeamSTEPPS produce the
expected outcomes?; (3) Was TeamSTEPPS worth
implementing in my unit, department, or work area?; and (4) If
TeamSTEPPS did not work or did not produce the expected
outcomes, why not? Not only should you and your colleagues—
as you collectively lead the charge to implement
TeamSTEPPS—care about the answers to these questions, but
so should your nursing home’s leadership.

Slide

As shown in the slide, measurement is important across all
phases of the TeamSTEPPS implementation process. The
results of measurement provide many benefits, including:
• Helping you identify where there are quality improvement
needs;
• Providing you with data to help generate leadership,
stakeholder, and/or staff buy-in of your efforts;
• Assessing training needs;
• Providing information to drive the plans for how you will use and
implement TeamSTEPPS;
• Providing an evaluation of the effectiveness of TeamSTEPPS
training;
• Assessing how TeamSTEPPS affects staff attitudes,
resident perceptions, and organizational culture; and
• Demonstrating successes and areas for continued
improvement, which is important for adapting implementation
plans.
The measures we will cover in this module will help you assess
your TeamSTEPPS implementation across the TeamSTEPPS
phases.

TeamSTEPPS for Long-Term Care | Measurement VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTION

D-10-5

Measurement

A MODEL OF TRAINING EVALUATION
SAY:

Slide

Kirkpatrick (1967) defined a multilevel model for evaluating the
impact of training programs. This model continues to be widely
used and regarded as a practical approach to training evaluation.
When determining the effectiveness of any training intervention,
Kirkpatrick advocated examining four different outcomes of
training:
1. Level I – Reactions – Reactions are defined as participants’
perceptions of the training. There are two types: (1) affective
reactions, which are related to whether participants “liked” the
training; and (2) instrumentality reactions, which are related to
whether participants found the training “useful.”
2. Level II – Learning – Learning is defined at three levels: (1)
attitudes (feel), (2) knowledge (know), and (3) skills (do).
Regarding attitudes, the basic question to be answered is, “Do
participants feel differently as a result of training?” Regarding
knowledge, the basic question to be answered is, “Do
participants know something new as a result of training?”
Regarding skills, the question is, “Can participants do
something differently/new as a result of training?”
3. Level III – Behavior – Behavior is defined as to whether the
new attitudes, knowledge, and/or skills are transferred to the
job. In other words, it measures whether participants use what
they learned in training on the job and whether that produces
improved job performance.
4. Level IV – Results – Results are defined as organizational
benefits that are produced from training. In the case of
TeamSTEPPS, results include resident outcomes, such as
the number of pressure ulcers and resident perceptions of
care, and clinical process outcomes, such as the number of
structured handoffs used and timeliness of medication
administration. The types of results depend on what the
TeamSTEPPS intervention targeted.

D-10-6

VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTION TeamSTEPPS for Long-Term Care | Measurement

AVAILABLE MEASURES

Measurement

SAY:
TeamSTEPPS provides a number of useful measures for
evaluating the impact of TeamSTEPPS. These measures are
aligned with the Kirkpatrick model of training evaluation.
• Reactions - TeamSTEPPS for Long-Term Care Course
Evaluation Form.

Slide

• Learning – TeamSTEPPS Teamwork Attitudes Questionnaire
(T-TAQ) for Long-Term Care (Attitudes), TeamSTEPPS
Learning Benchmarks for Long-Term Care (Knowledge), Team
Performance Observation Tool for Long-Term Care (Skills),
and TeamSTEPPS Teamwork Perceptions Questionnaire (TTPQ) for Long-Term Care (Skills).
• Behavior - Team Performance Observation Tool for LongTerm Care, TeamSTEPPS Teamwork Perceptions
Questionnaire (T-TPQ) for Long-Term Care, and AHRQ
Nursing Home Survey on Patient Safety Culture tools.
• Results – Resident outcomes and clinical process measures,
including the AHRQ Nursing Home Survey on Patient Safety
Culture tools and AHRQ Patient Safety Indicators.
We will review these measures in more detail and provide
information about where to access them.

TeamSTEPPS for Long-Term Care | Measurement VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTION

D-10-7

Measurement

REACTIONS: TEAMSTEPPS COURSE
EVALUATION FORM
SAY:

Slide

Participant reactions tell you whether participants liked the course,
the facilities, and the instructor, among other things. These types
of reactions are referred to as affective reactions. In addition,
participant reactions can tell you whether participants found
TeamSTEPPS useful and if they believe they c an apply the
information they have learned in their units, departments, or work
areas. These types of reactions are referred to as instrumentality
reactions. Research has shown that participants who find training
useful and report that they will use it on their jobs are more likely
to do so. Participant reactions also provide information regarding:
• What participants will say about the course to others who may
attend in the future; and
• Where the course could be improved in terms of what is taught
and how it’s taught.
If you plan to use a course evaluation form for continuing
education credit purposes, please check with your continuing
education provider to ensure that the form meets the specific
requirements of the respective accreditation organization.
A customizable TeamSTEPPS for Long-Term Care Course
Evaluation Form for assessing participant reactions can be
found in Tab F. This form captures both affective and
instrumentality reactions.

D-10-8

VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTION TeamSTEPPS for Long-Term Care | Measurement

LEARNING: TEAMSTEPPS TEAMWORK
ATTITUDES QUESTIONNAIRE (T-TAQ)

Measurement

SAY:
As noted, the evaluation of learning accomplished through training
can be in the form of assessing attitudes, knowledge, and skills.
One indication that training is effective is that participants’
attitudes about the importance of teamwork change as a result of
attending TeamSTEPPS training. To identify changes in attitudes,
participant attitudes should be measured both before and after
TeamSTEPPS training, with the expectation that participants’
scores will be higher after training. This would be an indication of
improved attitudes toward teamwork as a result of TeamSTEPPS
training.

Slide

The TeamSTEPPS Teamwork Attitudes Questionnaire, or T-TAQ,
for Long-Term Care, found in Tab F, was designed to measure
participant attitudes about the teamwork skills and behaviors
taught in TeamSTEPPS. The Long-Term Care instrument is a
customized version of the original T-TAQ, which has been tested
on numerous health care professionals. A report documenting the
original T-TAQ’s development, use, and interpretation can be
found on the AHRQ website.
For more information, go to:
http://teamstepps.ahrq.gov /abouttoolsmaterials.htm
It is very important to note that, with the explosion of the
resident safety movement and growing acceptance of the
importance of teamwork in the delivery of safe care, health care
professionals are likely to report positive attitudes toward
teamwork regardless of having attended TeamSTEPPS training.
Therefore, TeamSTEPPS recommends that you do not rely
solely on measuring participant attitudes as an indication of
learning.

TeamSTEPPS for Long-Term Care | Measurement

VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTION

D-10-9

Measurement

LEARNING: TEAMSTEPPS LEARNING
BENCHMARKS
SAY:

Slide

Another indication that training was effective is that participants
know something new after participating in training. Similar to
measuring attitude changes, measurement of participant
knowledge would take place both before and after TeamSTEPPS
training, using a knowledge test. The expectation would be that
participants would score higher on the test after training.
The Learning Benchmarks for Long-Term Care knowledge test
found in Tab F of the Instructor Manual can be used for this
purpose. The test is a short, multiple choice assessment that
measures participant knowledge of the teamwork principles
taught in TeamSTEPPS.
It is important to note that, just as attitude measures pose
challenges, so do knowledge measures. TeamSTEPPS has
found that despite careful construction of the Learning
Benchmarks, these items tend to be easy, and individuals are
often able to answer the items correctly without participating in
TeamSTEPPS training.

D-10-10 VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTIONTeamSTEPPS for Long-Term Care | Measurement

LEARNING: TEAM PERFORMANCE
OBSERVATION TOOL

Measurement

SAY:
The third indication that training was effective in terms of
participant learning is that participants can do something new
after participating in training. Similar to measuring attitudes and
knowledge, you would expect that if you observed participant
behavior in a simulation after TeamSTEPPS training, participants
would behave quite differently. For example, if you trained and
implemented SBAR and check-back as part of the medication
ordering procedure, you might test the resident care team by
developing a simulation to determine if learning occurred.
Observing the nursing staff placing orders to the pharmacy or
reconciling resident medication orders with the pharmacy on the
job could also be an indication of learning; but under the
Kirkpatrick hierarchy this is considered a measure of behavior—
learning transferred to the actual job.

Slide

The Team Performance Observation Tool for Long-Term Care is
a customizable tool that can be found in Tab F of the Instructor
Manual. This tool can be used as a guide when observing team
performance.
The Team Performance Observation Tool for Long-Term Care
included in TeamSTEPPS is generic, meaning it is not designed
to focus on any particular unit, department, or work area.
Therefore, TeamSTEPPS recommends that individuals:
• Customize the observation form to the unit, department, or
work area of interest.
• Practice using the tool prior to conducting observations. It is
important that anyone acting as an observer use the tool in the
same way so observations are accurate.
• Revise the observation tool as necessary so that information is
clear to all observers.

TeamSTEPPS for Long-Term Care | Measurement VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTION

D-10-11

Measurement

LEARNING: TEAMSTEPPS TEAMWORK
PERCEPTIONS QUESTIONNAIRE (T-TPQ)
SAY:

Slide

The TeamSTEPPS Teamwork Perceptions Questionnaire, or TTPQ, for Long-Term Care can also be used to measure the
learning of skills by training participants. The Long-Term Care
instrument is a customized version of the original T-TPQ, which
has been tested on numerous health care professionals.
The T-TPQ is completed by individual members of the team. In
this measure, individuals report their perceptions regarding the
effectiveness of the teamwork within the unit, department, or
work area in which they work. A copy of the T-TPQ for LongTerm Care is provided in Tab F. A detailed guide for using and
interpreting the original T-TPQ is available on AHRQ’s website.
For more information, go to:
http://teamstepps.ahrq.gov /abouttoolsmaterials.htm

D-10-12 VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTIONTeamSTEPPS for Long-Term Care | Measurement

BEHAVIOR

Measurement

SAY:
It is important to measure whether the information learned during
training is transferred to the job. Two important environmental
factors for producing transfer are:
1. Whether there is an opportunity to use the new
TeamSTEPPS tools or strategies on the job; and

Slide

2. Whether use of the TeamSTEPPS tools and strategies is
valued and reinforced.
Let’s consider an example.
Example:
As part of implementing TeamSTEPPS to improve teamwork
within the nursing home, all nurses are trained on the use of
SBAR for presenting resident information. However, physicians
are not invited to the training and receive no information about
the planned use of SBAR. Shortly after implementing the SBAR
strategy, a nurse calls a resident’s primary care physician about
a change in the resident’s health status. She begins to present
the resident’s situation and background, but before she can
complete her SBAR report, the physician jumps in and says, “I’m
not sure why you are presenting all of this background
information; can you just tell me what is wrong with Mrs. Smith?”
DISCUSSION
• Was the use of SBAR valued and reinforced?
• What do you think the nurse should do?
• W ill the nurse use SBAR in the future?
• What other factors can you think of that are important for
ensuring that skills learned in TeamSTEPPS training transfer
to the job?
- Aligning training objectives with nursing home goals
- Providing support for the training initiative
- Ensuring that frontline care leaders are on board
- Educating the residents’ care providers about SBAR and its
use within the nursing home
- Using measurement to determine the effectiveness of the
program in terms of whether skills are being used on the job
TeamSTEPPS for Long-Term Care | Measurement VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTION

D-10-13

Measurement

BEHAVIOR: MEASURES
SAY:
There are tools available for assessing the transfer of participants’
learning to their jobs. These include:
• The Team Performance Observation Tool for Long-Term Care;

Slide

• The TeamSTEPPS Teamwork Perceptions Questionnaire (TTPQ) for Long-Term Care; and
• AHRQ Nursing Home Survey on Patient Safety Culture.
We have already discussed the Team Performance Observation
Tool and the T-TPQ, as these can also be used to measure the
learning of new teamwork skills. The AHRQ Nursing Home
Survey on Patient Safety Culture can be used to assess both
behaviors and results. W e will first discuss a few other results
measures, and then focus our attention on a more detailed
discussion of the surveys.

D-10-14 VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTIONTeamSTEPPS for Long-Term Care | Measurement

Measurement

RESULTS: MEASURES
SAY:
The final level of training evaluation in the Kirkpatrick hierarchy is
results. As we discussed earlier, this level of evaluation provides
an assessment of the organizational benefits produced from
training.
It is important to note that this level of evaluation is difficult to
assess, in that any number of organizational changes, initiatives,
and/or interventions could contribute to organizational results.
However, for evaluating TeamSTEPPS, selecting measures that
align with the teamwork issue being addressed by TeamSTEPPS
will help ensure that the results can be linked to the training
intervention.

Slide

Results measures include:
• Resident outcome measures, such as pressure ulcers, falls,
pain, measurable medication errors, and the like, as well as
resident experience measures such as the Consumer
Assessment of Healthcare Providers and Systems (CAHPS)
Nursing Home Surveys.
• Clinical process measures, such as length of resident wait
time, medication administration delays, compliance with
infection control practices, compliance with treatment
protocols, and staff perceptions of safety as measured by
the AHRQ Nursing Home Survey on Patient Safety Culture.
It should be noted that health care facilities already routinely
collect many clinical quality and safety measures that can be
used to measure organizational results. These include, for
example, the CMS Quality Measures.

TeamSTEPPS for Long-Term Care | Measurement

VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTION D-10-15

Measurement

NURSING HOME SURVEY ON PATIENT SAFETY
CULTURE
SAY:

Slide

In response to nursing homes’ interest in a survey that focuses on
patient safety culture in their context, AHRQ sponsored the
development of the Nursing Home Survey on Patient Safety
Culture. This survey is designed specifically for nursing home
providers and staff, and asks for their opinions about the culture
of patient safety and healthcare quality in their nursing homes. It
is not designed for use in assisted living facilities, community care
facilities, or independent living facilities.
Many nursing homes using the AHRQ Nursing Home Survey on
Patient Safety Culture have expressed interest in comparing their
results to other nursing homes. In response, AHRQ has
established the Nursing Home Survey on Patient Safety Culture
Comparative Database. This database is a central repository for
survey data from nursing homes that have administered the
AHRQ nursing home culture survey instrument.
The survey is available from AHRQ on their website at:
http://www.ahrq.gov/professionals/quality-patientsafety/patientsafetyculture/nursing-home/index.html.
DISCUSSION
• Have any of you used the Nursing Home Survey on Patient Safety
Culture?
• Can you describe how you used the survey results?
• Do you have any lessons learned you would like to share?

Continued…

D-10-16 VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTION TeamSTEPPS for Long-Term Care | Measurement

NURSING HOME SURVEY ON PATIENT
SAFETY CULTURE (Continued)

Measurement

SAY:
The Nursing Home Survey on Patient Safety Culture has a
number of uses, including:
• Raising awareness about nursing home safety issues;
• Assessing resident safety culture;

Slide

• Tracking changes in resident safety culture over time; and
• Evaluating the impact of resident safety interventions.
The nursing home survey can be used to assess whether
TeamSTEPPS has a positive impact on the nursing home’s
resident safety culture. If TeamSTEPPS is effective, survey
scores should increase after TeamSTEPPS tools and
strategies have been implemented. Such a result would be
considered an organizational outcome or result of the training
(i.e., Level IV in the Kirkpatrick hierarchy).

Continued…
TeamSTEPPS for Long-Term Care | Measurement VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTION

D-10-17

Measurement

NURSING HOME SURVEY ON PATIENT
SAFETY CULTURE (Continued)
SAY:
The nursing home survey includes 42 items that represent 13
dimensions.

Slide

• Eleven dimensions that are important to resident safety, such
as teamwork, compliance with procedures, handoffs, and
communication openness;
• One dimension related to overall perceptions of resident safety
in the nursing home; and
• One overall judgment about the nursing home.
The survey takes approximately 10-15 minutes to complete. Most
of the items use Agree/Disagree or Never/Always response
categories, making them easy to answer. Room for written
comments is provided at the end of the survey.

Continued…
D-10-18 VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTION TeamSTEPPS for Long-Term Care | Measurement

NURSING HOME SURVEY ON PATIENT
SAFETY CULTURE (Continued)

Measurement

SAY:
The Nursing Home Survey on Patient Safety Culture has an
accompanying toolkit that contains the following materials:
• Survey forms;
• Survey items and dimensions;
• A Survey User's Guide that provides step-by-step instructions on
how to select a sample, administer the survey, and analyze and
report results; and

Slide

• A survey feedback report template in Microsoft® PowerPoint,
which can be customized to display survey results to
administrators and staff throughout the nursing home and for
presentation purposes.
In addition, there is a data entry and analysis tool that works
with Microsoft® Excel and makes it easy to input your individuallevel data from the survey. The tool then automatically creates
tables and graphs to display your survey results.
To request the tool,email:
[email protected].
If you have questions about the Surveys on Patient Safety
Culture, or need technical assistance, you can request help by
email to
[email protected] or
[email protected].

TeamSTEPPS for Long-Term Care | Measurement VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTION

D-10-19

Measurement

EXERCISE: EVALUATING TEAMSTEPPS
SAY:
Now that we have reviewed the four levels of the Kirkpatrick
model of training evaluation and discussed available measures for
each, we will use a scenario to apply what you’ve learned. Our
scenario follows:

Slide

Slide

TIME:
10 minutes
0 MATERIALS:
• 0Evaluating

TeamSTEPPS
Worksheet

Staff from the dementia care unit recently attended the 2-day
TeamSTEPPS Master Training course. During the
implementation planning session, the team noted that Mr.
Stevens has been increasing instances of coming out of his
room in just his socks and yelling obscenities during the
evening shift change. When this occurs, one of the nurses
finds Mr. Stevens, assesses the situation, and tries to get Mr.
Stevens settled down and back in his room. Many times
additional nurses are required, but the shift change has led to
some confusion about who should help. Further, because Mr.
Stevens’ room is not visible from the area where the shift
change occurs, it is often unclear about whether and how
much help the responding nurse might need. Mr. Stevens also
becomes more agitated if the nurse talking with him calls for
help, causing one or more additional staff to arrive.
To help keep Mr. Stevens calm while ensuring that the
responding nurse has support when needed, the team decides
to use CUS as a signal to call for additional help with Mr.
Stevens. Because they can hear what is going on in his room
from the area where the evening shift change briefing occurs,
they agree that if they hear a staff person using CUS with Mr.
Stevens, it will result in two additional staff going to provide
assistance with Mr. Stevens. They also agree that prior to
conducting the shift change briefing, the staff will discuss Mr.
Stevens, identify who will initially respond to any disruption,
and determine who will provide additional assistance if
needed.
Break into small groups and identify (1) one or two measures you
can use to assess whether your TeamSTEPPS intervention –
CUS– transfers to the dementia unit; and (2) one or two
measures to determine if the use of CUS affects resident
outcomes.

Continued…
D-10-20 VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTION TeamSTEPPS for Long-Term Care | Measurement

EXERCISE: EVALUATING TEAMSTEPPS
(Continued)

Measurement
Slide

DO:
After a few minutes, reconvene the participants and ask some
groups to report on their identified measures, using the questions
presented on the following page.

DISCUSSION:
• Which measure or measures did you identify for assessing
whether instruction about CUS transfers to the dementia
unit’s work environment?
• Why did you choose the identified measures?
• What organizational barriers might interfere with successful
training transfer and how can these barriers be addressed?
• Which measure or measures did you identify for determining
whether CUS affects resident outcomes?
• Why did you choose the identified measures?

TeamSTEPPS for Long-Term Care | Measurement VERSION FOR REVIEW ONLY; NOT FOR DISTRIBUTION

D-10-21