Supporting Statement Part B -- Improving Patient Safety System Implementation for Patients with LEP 5 10 11

Supporting Statement

Part B

Improving Patient Safety System Implementation for Patients with Limited English Proficiency

May 6th, 2011

Agency for Healthcare Research and Quality

Table of contents

1. Respondent universe and sampling methods 3

2. Information Collection Procedures 3

3. Methods to Maximize Response Rates 6

4. Tests of Procedures 6

5. Statistical Consultants 7

Part B. Collections of Information Employing Statistical Methods

1. Respondent universe and sampling methods

Three hospital sites will participate in this project.  Each site will be the object of a case study.  The sites were pre- selected using purposive sampling to demonstrate whether the training module can be implemented in hospitals with differing characteristics including different geographic locations, sizes, LEP populations, TeamSTEPPS experience levels and resource levels.  Eligibility criteria for facilities include the willingness of a hospital leader to participate in the project and presence of a sizable patient population with limited English proficiency.  Sites were selected from among the 134 hospitals that have participated in the Disparities Leadership Program.  The Disparities Leadership Program is a year-long executive education program designed for leaders from hospitals, health plans and other health care organizations who want to develop a strategic plan or advance a project to eliminate racial and ethnic disparities in health care.

Facilities were informed of the opportunity to participate by an email sent to past and present participants in the Disparities Leadership program.  Interested hospitals were asked to send information about their hospital including hospital location, type, LEP populations served, and prior experience with TeamSTEPPS.  Based on email responses received we selected three hospitals to participate:

1. Christiana Care of Newark, DE               (TeamSTEPPS site)

2. HealthEast of Woodbury, MN                (TeamSTEPPS site)

3. Children's Mercy of Kansas City, MO (non-TEAMSTEPPS site)

Exhibit 1, below, provides a summary of these hospitals’ characteristics. Should any site choose not to participate in the study, we will seek a suitable replacement site from among other interested hospitals that responded to our email inquiry.  If need be, the pool of replacement sites may be broadened to include hospitals participating in the TeamSTEPPS national implementation collaborative but not in the Disparities Leadership program.

We make no claim that the results from this study will be generalizable in the statistical sense. Rather, our small sample of information-rich cases will be illustrative and informative and will generate lessons that will inform adjustments to training materials and measurement instruments for the new TeamSTEPPS training module on patients with limited English proficiency.

Hospital	Location	Type	Team STEPPS site?	Patient Race/Ethnicity/ Languages	Number of Beds
Christiana Care Health System	Newark, DE	Non-profit, privately owned teaching hospital	Yes	26% African American, 7% Hispanic, 1% Other, 66% Caucasian. The most common non-English language spoken by patients is Spanish.	1100
HealthEast	Woodbury, MN	Non-profit, faith-based health care organization	Yes	Asian, 17.8%, African American 7.2%, Hispanic 3.5%, Indian American 0.7%, Other 1%, Pacific Islander 0.2% and unidentified 1.2%. Common languages include sign language, Hmong, Spanish, Somali, and Karen.	St Joseph’s hospital: 258; Bethesda Hospital: 142; Woodwinds: 86; St John’s, 145
Children’s Mercy Hospital	Kansas City, MO	Non-profit, urban safety net hospital	Not yet	Asian 1.2%, Black 24.2%, non-Hispanic White 56.5%, Hispanic 12.3%, Native American 0.1% Other 5.6%, not specified 0.1%. The top 4 languages are Spanish, Somali, Arabic, and Vietnamese.	315 (between 2 hospitals with ~270/50 division)

Exhibit 1. TeamSTEPPS Sites Characteristics

2. Information Collection Procedures

Sample Size

Number of case studies

Budget constraints will limit the number of field test sites to three. While it is always preferable to have a larger number of test sites, we anticipate that three sites will provide us with sufficient information to identify needed revisions to the LEP module’s content and format.

Sample size within each case study

• Staff surveys:

Each participating hospital will purposively select one or more units (e.g., Ob/gyn, Emergency Department) to receive the TeamSTEPPS LEP training. All staff working in the selected unit(s) including doctors, nurses, technicians and interpreters, will be eligible to participate in the training. Typically this results in a training group size of approximately 15 to 30 staff members for each participating unit. This is the largest sample size hospitals might reasonably be willing to mobilize for field test of this type. Trainees will be administered the questionnaires described in Attachments F through I.

Proposed analyses are described further below. If analyses conducted at each hospital fail to show statistically significant results, AHRQ may conduct a pooled data analysis to attain a sample size of 45-90 staff persons. Details of statistical power calculations for the proposed analyses are provided further below.

• Patient surveys:

Each participating hospital will also be asked to conduct a patient survey with a sample of 30 LEP patients before and after the training (60 patients total). Hospitals will be asked to use a systematic method to select patients with limited English proficiency for this survey (for example, hospitals might select every LEP patient seen, or every second or third patient seen, over a given period or until they attain the desired sample size). The survey is currently available in Spanish, which is the most commonly spoken non-English language. To assure that the sample selected is representative of each hospital’s LEP population, hospitals will be asked to offer the survey to all LEP patients through their standard language assistance mechanisms.

Proposed analyses are described further below. If analyses conducted at each hospital fail to show statistically significant results, AHRQ may conduct a pooled data analysis to attain a sample size of up to 180 patients (90 pre- and 90 post-training). Details of statistical power calculations for the proposed analyses are provided further below.

Proposed analyses and statistical power calculations

As described in part A of this application, most of the data collected will be used for descriptive purposes and analyzed using qualitative case study methods. Quantitative analyses will complement case study analyses. They will include univariate statistics (e.g., average trainee satisfaction score) and, where appropriate, statistical tests to assess the differences between pre- and post-training scores.

Variables for which pre- and post-test differences will be analyzed are listed in exhibit 1, below. To test pre-post differences, we will use t-tests for continuous variables and binomial tests for dichotomous variables. Variables with more than two categories will be recoded into two categories so that binomial tests can be performed. A paired t-test will be used to test differences in average learning scores before and after training. All other tests will be unpaired to keep hospitals’ data collection burden within the bounds of what they are likely willing to do.

Exhibit 2, below, provides details of the variables on which statistical tests may be conducted, proposed tests, sample sizes, and minimum size of differences that can be detected at an 80% statistical power level given the sample size and explicit assumptions about the standard deviation (for t-tests) and pre-test proportions (for binomial tests).

Inclusion and selection criteria

As described above, hospitals were pre- selected based on their willingness to participate and based on the size of their patient populations with limited English proficiency.

To assess whether the training module can be implemented in a variety of different geographic settings and in different types of hospitals, we pre-selected three hospitals that are as different from one another as possible on several dimensions including size, resource level, geographic location, and whether they or not they have a TeamSTEPPS program (see Exhibit 1, above). Should one of these sites decide withdraw from the study, a suitable replacement will be found.

Within each site, hospitals will designate two trainers and select one or more units that will receive the proposed training. All staff members in that unit, and the interpreters who work with them, will be eligible for and invited to the training.

Exhibit 2. TEAMSTEPPS LEP Module sample size and statistical power calculations

Instrument	Variables used	Statistical test	Assumed Sample size	Minimum detectable effect size at 80% statistical power level and 5% alpha
Learning Outcomes Survey (Attachment G)	Average scores pre- and post-training (range: 0-5)	Paired t-test to compare pre- and post-test scores.	Per hospital: 16 persons pre- and post Pooled: 48	Paired T-tests assuming sd=1 Per hospital: 0.75 point change assuming sd=1 Pooled: 0.42 point change assuming sd=1
Behavior Surveys (Attachments H and I)	Pre- and post-test scores on use of interpreter services (range: 0-100%) Ordinal variables recoded to a binary form	Unpaired t-test to compare pre- and post-test scores on use of interpreter services Binomial test on binary variables	Per hospital: 14 persons pre- and 12 post (accounts for exclusion of interpreters and attrition). For paired samples N=12 Pooled: 42 pre and 36 post. For paired samples N=36	Paired T-tests Assuming sd=10 Per hospital: 8.9 percentage point change Pooled:4.8 percentage point change Binomial tests Starting proportion=20% Per hospital: 59percentage points Pooled: 33 percentage points
Patient Outcome Survey (Attachment J)	Question 19 about overall satisfaction (range: 0-10) One to five ordinal variables of interest to each hospital. Ordinal variables will be recoded to binary form.	Unpaired t-test for Question 19 Binomial test on all other variables	Per hospital: 30 persons pre- and post Pooled: 90	Unpaired T-tests assuming sd=10 Per hospital: 7.5 percentage point change Pooled:4.2 percentage point change Binomial tests Assuming starting proportion=20% Per hospital: 30 percentage points Pooled:17 percentage points

3. Methods to Maximize Response Rates

Hospital participation will be maximized by offering free training, travel to the training site for trainers, and by offering technical assistance with implementation. Response rates for the hospital staff surveys will be maximized by encouraging hospitals to collect as much data as possible while trainees are in the room. One survey component (survey of staff behaviors after the training) may be collected electronically if site liaisons feel it is appropriate to their setting. In this case, the respondent will be given a visual representation of their percent completion of the survey to maximize response rates. For the patient survey response rate will be maximized by keeping the survey short, and letting patients know before their medical visit that they will be asked to complete a survey when they exit, Some hospitals may also choose to offer small incentives to survey respondents, however this is not strictly necessary.

4. Tests of Procedures

Trainee satisfaction and learning outcome surveys follow the format of TeamSTEPPS measurement instruments, though the content has been adapted to this specific training module. Survey items for the patient outcome survey were drawn from a previously validated HCAHPS (Hospital Consumer Assessment of Healthcare Providers and System) survey developed by AHRQ.

The behavior survey is a new type of instrument for TeamSTEPPS focusing on the behaviors that the module is intended to modify. That survey has not been previously tested. The field test will help us to assess whether hospitals might be willing to implement such a survey. Further testing of this instrument may be considered for a later data collection effort if this minimum threshold is met.

5. Statistical Consultants

Abt Associates is the contractor who will facilitate hospitals’ data collection and analysis on behalf of AHRQ. Abt Associates specializes in public policy and opinion surveys, banking and finance, telecommunications, media, energy, transportation, insurance and health care. The professionals from Abt Associates have over 40 years of experience providing high quality, timely and cost effective surveys for federal, state, local, and private clients. They conduct over 400 surveys each year, which involve meeting a wide variety of data entry, editing and transfer specifications. The key contact at Abt Associates is Melanie Wasserman.

Contact information for Dr. Wasserman is provided below.

Melanie Wasserman, Ph.D.

[email protected]

Abt Associates, Inc.

55 Wheeler St.

Cambridge, MA 02138

Phone: (617) 520-2714

Fax: (617) 386-7696

8