AHRQ Memo 7-7-09

7/7/09

Responses to further concerns (7/2/09) about AHRQ-Lean Implementation

1. This study is seeking to assess patient satisfaction from the perspective of the organization. Will there be a press-ganey survey for patients as there was in the ED ICR to more directly assess patient satisfaction?

Response:

For the Lean implementation project, we do not require organizations to collect the Press-Ganey survey. Improving patient satisfaction is not often an explicit objective of Lean projects. Nonetheless, if the participating organizations collect satisfaction data through a vendor as part of their normal business procedures, and they are willing to share the data, we will use the data in our study (see Question 2c below).

2. We are fine with adding case studies and are actually pleased that a prospective arm will be added to each (with the exception of the Virtua Lean hospital implementation, which will be entirely prospective). However, we have a few questions:

a. what is the sequence of the retro and prospective arms and how are they related? And will the retro precede the prospective (e.g. you do the retro and use the prospective to address any data gaps)?

Response:

The retrospective case studies and the first site visit of the prospective case studies will be conducted simultaneously. Each organization serves as the site for one retrospective and one prospective case study. The interviews with senior management (e.g., CEO, CFO, CIO, etc.) will be used for both the retrospective and prospective cases (i.e., they will occur once). In circumstances where scheduling difficulties make it impossible to conduct the site visits for both types of studies simultaneously, we will try to collect data in as concurrent a time frame as possible.

The retrospective and prospective case studies will target different departments and different projects. Thus, the prospective study will not address data gaps of the retrospective study.

b. to the extent that there will be prospective arms to these studies, it seems like you would want some form of baseline data collection and then at least one further follow-up data collection. Where is the baseline data going to come from? You did include some topic guides for use in prospective studies, but these seem to be implemented 3 months into the study. It seems like AHRQ would get more reliable/credible data if the baseline data collection were initiated at the very beginning of the study and prior to the intervention being implemented. Since part of this study is assessing how sites pick the "tools" they want to use, it seems like it would be valuable to get an understanding at the beginning of why they picked the tool, what they expect to gain from it, and then see how these responses change over time.

Response:

The retrospective and prospective case studies will occur in the same organization, but as mentioned in our response to 2a, within different departments. For each prospective case study, during the first site visit we will collect data just before or just after the intervention is initiated. In other words, the first site visit will occur immediately before, on, or immediately after the start of Lean implementation in the department. In either case, we will gather valuable information on the reasons why sites chose specific tools, what they hoped to and actually gained, and how, if at all, their perspectives about the tools and Lean/TPS have changed over time.

In addition, the aim of this study is not to evaluate a single technique or intervention. Instead, using a purposive sample of organizations and projects, we are capitalizing on a series of natural experiments to describe and assess both the similarities and differences in the ways in these sites have implemented Lean/TPS, related challenges and solutions, and to assess whether a compelling argument might be made for the value of Lean/TPS in health care, and for whom. Since, our study objectives are exploratory and our design is neither an experiment nor a quasi experiment, our data from the first site visits will not serve as baseline data. We will use the data from the site visits and the additional data we gather to draw conclusions on the evidence and on factors affecting the overall success of Lean across the case studies.

c. to the extent that there are prospective arms to these studies, it seems like AHRQ could provide some requirements on the types of data each site is required to collect. On the in-person topic guide, for example, one of the questions is "In your opinion, is there a case for Lean implementation..." and "what kind of data do you collect?" One of the possible response categories appears to be "none." As long as you're doing prospective studies, it seems to make sense to specify for each site what type of data they are going to be required to collect, especially on the areas listed on page 8 (e.g. efficiency, patient safety, etc.).

Response:

To the extent possible, we will collect similar data from the organizations that are voluntarily participating in the study. However, we will focus on the organization’s routinely collected outcomes and process data, which are collected as part of normal business practices or quality improvement activities. We are asking organizations to only provide data that they are already collecting for three reasons:

First, a goal of our study is to learn about the data organizations are independently collecting to assess the impact of Lean. By learning more about the types of data and measures collected and their value and contribution to understanding the impact of Lean/TPS, we will be able to make recommendations for data and measures that organizations should collect.

Second, for the above reason, the RFTO and subsequent project modification agreements do not include funding for primary or consistent data collection and do not plan for compensation to the sites. Thus, any request or requirement for consistent data collection would impose a significant burden on participating organizations due to the costs in terms of staff resources and time to gather the data.

Third, imposing such a requirement could result in selection bias in our sample of organizations towards organizations with the largest resources, and possibly the most sophisticated electronic health data, are the most likely to be able to supply the data without new collection efforts.

d. to make it more clear which questions will be used for retro studies and which for prospective, which for baseline and which for follow-up, could we break up the topic guides into separate instruments?

Response:

The questions in the topic guides will be used for both the retrospective and prospective case studies. The only difference is that the tense of the questions will be altered to fit the analytic perspective. The retrospective case study in-person interviews will include the same questions as the combination of the first and second prospective case study in-person interviews. For example, in the retrospective in-person interviews, we will ask about how the results from the Lean projects have been disseminated, using the past tense. In the second site visit as part of the prospective case study, we will ask the interviewees how they are disseminating the findings, using the present tense.

3. study sites

a. most of the sites selected appear to be large, urban, hospitals, and early adopters. Seems like we need some more diversity (i.e. smaller hospitals/clinics and late adopters). Also, one of the dimensions listed in part B was "linear vs. non-linear" LEAN. How do the sites vary on that dimension?

Response:

Including key dimensions of variation in sites for this study are crucial. Although unclear in our first submission, the site most likely to be selected from the Mayo Health System is the Luther clinic in Wisconsin. This site is neither large nor urban. In addition, we have two sites which are to-be-determined. We will ensure that the new sites selected will increase the diversity of sampled organizations. Finally, each site will include one case study focusing on linear Lean implementation, and one case study focusing on non-linear Lean implementation case study.

3