Supporting Statement Part B -- HSOPS Comparative Database 7-31-2013

SUPPORTING STATEMENT

Part B

Collection of Information for

Agency for Healthcare Research and Quality’s (AHRQ)

Hospital Survey on Patient Safety Culture Comparative Database

Version April 30, 2013

Agency of Healthcare Research and Quality (AHRQ)

Table of contents

B. Collections of Information Employing Statistical Methods 3

1. Respondent universe and sampling methods 3

2. Information Collection Procedures 11

3. Methods to Maximize Response Rates 14

4. Tests of Procedures 15

5. Statistical Consultants 15

B. Collections of Information Employing Statistical Methods

1. Respondent universe and sampling methods

Universe of hospitals and representativeness of the data. The AHRQ Hospital Survey on Patient Safety Culture Comparative Database serves as a central U.S. repository for data on the survey and AHRQ houses the largest database of the survey’s results. However, the database is comprised of data that are voluntarily submitted by hospitals that have administered the survey, and is not a statistically selected sample, nor is it a representative sample of all U.S. hospitals.

An AHRQ Hospital Survey on Patient Safety Culture Comparative Database Report has been produced yearly since 2007 (there are reports from 2007 through 2012 on the AHRQ web site at http://www.ahrq.gov/legacy/qual/patientsafetyculture/hospsurvindex.htm). At the end of Chapter 1 in the Comparative Database Report is a section entitled “Data Limitations” that outlines the limitations of the data. The section makes clear that the data are not a statistically selected sample of U.S. hospitals; the hospitals were not required to undergo training on survey administration and have administered it using different modes; some hospitals conducted a census of all staff while others surveyed a sample of staff; and that basic data cleaning for out-of-range values was conducted on the database data but are not otherwise verified as accurate.

For the 2012 database, participating hospitals administered the Hospital Survey on Patient Safety Culture to their hospital staff between January 2008 and June 2011 and voluntarily submitted their data for inclusion in the database. Because hospitals do not necessarily administer the hospital patient safety culture survey every single year, but may administer it on an 18-month, 24-month, or other administration cycle, the comparative database is a “rolling” comparison. Data from prior years is retained in the database when a hospital does not have new data to submit; older data is replaced with more recent data when it is available; and new data is added from hospitals submitting for the first time.

Table B-1a and B1-b present the 1,128 hospitals in the 2012 Comparative Database categorized by whether they were previous submissions or 2011 submissions, and whether they had only submitted once or more than once.

Table B-1a. Overall Statistics for the 2012 Database Participating Hospitals

Overall Statistic	Retained From the 2011 Database	Submitted for the 2012 Database	Total 2012 Database
Number of hospitals	508	620	1,128
Number of individual survey respondents	263,543	304,160	567,703

Table B-1b. Statistics for Nontrending and Trending Hospitals in 2012 Database

Overall Statistic	Nontrending (Submitted Once)	Trending (Submitted More Than Once)	Total 2012 Database
Number of hospitals	478	650	1,128
Number of individual survey respondents	218,167	349,536	567,703

The number of hospitals in the U.S. is estimated to be 5,795 (U.S. Census Bureau:http://www.census.gov/compendia/statab/cats/health_nutrition/health_care_resources.html, Table 172 Hospitals; accessed April 8, 2013.) However, the latest AHRQ Hospital Survey on Patient Safety Culture Comparative Database Report consists of data from 1,128 hospitals which represents only 19.5% of the total estimated population of U.S. hospitals.

In an effort to discern the comparability of the 1,128 hospitals in the 2012 Database Report to hospitals in the population, AHRQ presents statistics comparing the characteristics of the 1,128 hospitals (bed size, teaching status, ownership & control, region) against characteristics of U.S. hospitals based on those registered with the American Hospital Association (AHA). Although the AHA data set, which is updated every 2 years, does not capture characteristics on the entire population of U.S. hospitals, it is used as the comparison because it is the best source of data on hospital characteristics and includes more hospitals than any other data set.

Comparisons of the 1,128 database hospitals against characteristics obtained from the AHA (the 2006 or 2010 AHA Annual Survey of Hospitals Database © Health Forum, LLC) are provided in Tables B-2 to B-5 and are displayed on pages 16 to 17 in Chapter 3 of the 2012 Comparative Database Report (included as Attachments A and B) and available on the AHRQ legacy web site at http://www.ahrq.gov/legacyqual/hospsurvey09/).

The tables show that the 1,128 database hospitals are similar to the distributions of characteristics from AHA-registered hospitals. However, the database overrepresents larger hospitals, teaching hospitals, and nongovernment hospitals. In addition, the database distribution under-represents Mid Atlantic/New England and West South Central hospitals, and over-represents the East North Central and West North Central hospitals compared to the distribution of AHA-registered U.S. hospitals.

Table B-2. Distribution of Database Hospitals and Respondents by Bed Size

(Compared to AHA-registered U.S. Hospitals)

Bed Size	AHA-registered U.S. Hospitals		2012 Database Hospitals		2012 Database Respondents
	Number	Percent	Number	Percent	Number	Percent
6-24 beds	657	10%	74	7%	7,322	1%
25-49 beds	1,418	22%	165	15%	22,687	4%
50-99 beds	1,347	21%	196	17%	47,914	8%
100-199 beds	1,326	21%	250	22%	94,361	17%
200-299 beds	709	11%	192	17%	120,566	21%
300-399 beds	409	6%	94	8%	72,147	13%
400-499 beds	218	3%	63	6%	68,752	12%
500 or more beds	323	5%	94	8%	133,954	24%
TOTAL	6,407	100%	1,128	100%	567,703	100%

Table B-3. Distribution of Database Hospitals and Respondents by Teaching Status

(Compared to AHA-registered U.S. Hospitals)

Teaching Status	AHA-registered U.S. Hospitals		2012 Database Hospitals		2012 Database Respondents
	Number	Percent	Number	Percent	Number	Percent
Teaching	1,516	24%	386	34%	322,030	57%
Non-Teaching	4,891	76%	742	66%	245,673	43%
TOTAL	6,407	100%	1128	100%	567,703	100%

Table B-4. Distribution of Database Hospitals and Respondents by Ownership and Control

(Compared to AHA-registered U.S. Hospitals)

Ownership and Control	AHA-registered U.S. Hospitals		2012 Database Hospitals		2012 Database Respondents
	Number	Percent	Number	Percent	Number	Percent
Government (federal or non-federal)	1,645	26%	229	20%	109,656	19%
Non-Government (voluntary/non-profit or proprietary/investor owned)	4,762	74%	899	80%	458,047	81%
TOTAL	6,407	100%	1,128	100%	567,703	100%

Table B-5. Distribution of Database Hospitals and Respondents by Geographic Region

(Compared to AHA-registered U.S. Hospitals)

Region	AHA-registered U.S. Hospitals			2012 Database Hospitals				2012 Database Respondents
	Number	Percent		Number			Percent	Number	Percent
New England	602	9%		82			7%	70,275	12%
Mid Atlantic	271	4%		30			3%	23,838	4%
South Atlantic/Associated Territories	1,016	16%		182			16%	101,597	18%
East North Central	925	14%		279			25%	142,124	25%
East South Central	533	8%		110			10%	36,841	6%
West North Central	803	13%		142			13%	47,469	8%
West South Central	1,089	17%		139			12%	56,740	10%
Mountain	509	8%		73			6%	39,543	7%
Pacific/Associated Territories	659	10%		91			8%	49,276	9%
TOTAL	6,407	100%		1,128			100%	567,703	100%
NOTE: States are categorized into AHA-defined regions as follows:
New England: ME, NH, VT, MA, RI, CT Mid Atlantic: NJ, NY, PA				West North Central: MN, IA, MO, ND, SD, NE, KS
South Atlantic: DE, MD, DC, VA, WV, NC, SC, GA, FL				West South Central: AR, LA, OK, TX
East North Central: OH, IN, IL, MI, WI				Mountain: MT, ID, WY, CO, NM, AZ, UT, NV
East South Central: KY, TN, AL, MS				Pacific: WA, OR, CA, AK, HI

Increase in the number of hospitals to be included in the 2014 Comparative Database Report. In January 2009, the Joint Commission, which is a hospital accreditation organization, enacted a requirement for hospitals to regularly evaluate the culture of safety and quality as part of its Hospital Leadership Standards (Standard LD.03.01.01, http://www.jointcommission.org/Standards/ or view a pre-publication version of their 2009 Standards which reference the assessment of safety culture on page 9 of 25: http://www.jointcommission.org/NR/rdonlyres/D53206E8-D42B-416B-B887-491B6D5AA163/0/HAP_LD.pdf accessed on June 24, 2009.) Due to this requirement, the number of hospital data submissions increased from 535 hospitals in 2009 to 1,128 hospitals in 2012 . It is anticipated that due to this Joint Commission requirement, more hospitals will administer the AHRQ survey in subsequent years and the database will continue to grow by a significant rate.

Universe of individual hospital staff respondents and representativeness. The data submitted by hospitals are individual hospital staff respondent data from the AHRQ Hospital Survey on Patient Safety Culture. The 2012 database consists of data from 196,462 hospital staff respondents across 1,128 participating hospitals.

As part of the process of data submission, hospitals are asked to provide data about survey administration. The self-reported survey administration statistics shown in Tables B-6, 7, and 8 are provided in Chapter 2 of the Comparative Database Report (on pages 13-14).

Table B-6. Summary Statistics for 2012 Database Participating Hospitals

Average Number of completed surveys per hospital (range: 12 to 8,725)	503
Average Number of surveys administered per hospital (range: 15 to 13,000)	1,182
Average hospital response rate (range: 4% to 100%)	53%

Table B-7. Survey Administration Statistics

Survey Administration Mode	2012 Database Hospitals		2012 Database Respondents
	Number	Percent	Number	Percent
Paper only	239	21%	55,194	10%
Web only	746	66%	424,366	75%
Both paper and web	143	13%	88,143	16%
TOTAL	1,128	100%	567,703	100%

Table B-8. Average Hospital Response Rate by Mode

Survey Administration Mode	Average Hospital Response Rate
Paper only	61%
Web only	51%
Both web and paper	49%

Most hospitals (1,014 or 90%) administered the survey to a census of all hospital staff, or a sample of staff, from all hospital work areas/units; fewer hospitals (94 or 7%) administered the survey to a subset of selected staff or work areas/units; and 20 hospitals (2 percent) administered the survey to a subset of selected staff and selected work areas/units (see Table B-9).

Table B-9. Types of Staff or Work Areas/Units Surveyed

Types of Staff or Work Areas/Units Surveyed	2012 Database Hospitals		2012 Database Respondents
	Number	Percent	Number	Percent
All staff, or a sample of all staff, from all work areas/units	1,014	90%	533,915	94%
Selected staff only	63	6%	18,918	3%
Selected work areas/units only	31	3%	7,137	1%
Selected staff and selected work areas/units	20	2%	7,733	1%
TOTAL	1,128	100%	567,703	100%

Basic characteristics of individual hospital respondents such as their primary work area, staff position, and their direct interaction with patients are provided in the Comparative Database Report (pages 19 and 20) and shown in Tables B-10 to B-12.

Table B-10. Distribution of Database Respondents by Work Area/Unit

Work Area/Unit	2012 Database Respondents
	Number	Percent
Other	159,889	30%
Surgery	52,072	10%
Medicine	62,688	12%
Many different hospital units/No specific unit	44,024	8%
Intensive care unit (any type)	36,402	7%
Radiology	30,215	6%
Emergency	30,111	6%
Laboratory	25,705	5%
Obstetrics	23,917	4%
Rehabilitation	18,293	3%
Pharmacy	15,725	3%
Pediatrics	16,976	3%
Psychiatry/mental health	15,600	3%
Anesthesiology	3,538	1%
TOTAL	535,155	101%
Missing: Did not answer or were not asked the question	32,548
Overall total	567,703

Table B-11 Distribution of Database Respondents by Staff Position

Staff Position	2012 Database Respondents
	Number	Percent
Registered Nurse (RN) or Licensed Vocational Nurse (LVN)/ Licensed Practical Nurse (LPN)	191,402	35%
Other	114,425	21%
Technician (EKG, Lab, Radiology)	58,495	11%
Administration/Management	42,021	8%
Unit Assistant/Clerk/Secretary	34,782	6%
Patient Care Asst/Hospital Aide/Care Partner	29,726	6%
Therapists (Respiratory, Physical, Occupational or Speech)	25,160	5%
Attending/Staff Physician, Resident Physician/ Physician in Training, or Physician Assistant (PA)/Nurse Practitioner (NP)	29,832	6%
Pharmacist	10,339	2%
Dietician	3,332	1%
TOTAL	539,514	101%
Missing: Did not answer or were not asked the question	28,189
Overall total	567,703

Table B-12. Distribution of Database Respondents by Interaction with Patients

Interaction With Patients	2012 Database Respondents
	Number	Percent
YES, have direct patient interaction	407,884	76%
NO, do NOT have direct patient interaction	130,135	24%
TOTAL	538,019	100%
Missing: Did not answer or were not asked the question	29,684
Overall total	567,703

Comparative results and explanation of how results are calculated. Using data from the database hospitals, the Comparative Database Report presents average percent positive scores for each of the 12 patient safety culture composites and for the survey’s 42 items (plus two additional questions on patient safety grade and number of events reported). The average percent positive scores were calculated by averaging composite-level percent positive scores (which, in turn, are average percentages of positive response to the items in a composite) across all hospitals in the database, as well as averaging item-level percent positive scores across hospitals. Since the percent positive is displayed as an overall average, scores from each hospital are weighted equally in their contribution to the calculation of the average.¹

Percentages are presented rather than mean scores because hospital administrators have indicated that percentages are more easily understood and interpreted (for example, indicating that 75% of staff responded positively to an item rather than reporting that the mean score on the item was 4.00 out of 5). In addition, the minimum and maximum percent positive scores are presented along with percentile scores for the 10^th, 25^th, 50^th, 75^th, and 90^th percentiles to present information about the distribution of scores across database hospitals.

Trending results are also presented in the Comparative Database Report (pages 43- 58) and are based only on the subset of hospitals that have submitted data more than once. Trending results display the average percentage of positive response for previous and most recent administrations of the survey, the change in scores (as a difference between the average scores), the maximum increase and decrease in scores, and the average increase and decrease in percent positive scores over time.

Guidance to hospitals on how to compare their results against the database. Hospitals that submit data to the database receive a free, individual, customized feedback report that displays the hospital’s results against the database. Hospitals that do not submit data to the database can still compare their results to the database. The last section of the Comparative Database Report “Notes: Description of Data Cleaning and Calculations” (pages 65-69), provides instructions on how to calculate percent positive scores to enable hospitals to calculate their own scores to compare their results against the database. As part of a toolkit of support materials for the Hospital SOPS survey, hospitals can use a Microsoft® Excel-based Data Entry and Analysis Tool that is an Excel file with macros that will automatically produce graphs and charts of a hospital’s results once data are entered into a data sheet. Many hospitals use this tool to produce their results.

In the chapter “Comparing Your Results” (beginning on page 30 in the 2012 Report), hospitals are provided with a detailed description and explanation of the statistics that are presented and given examples and guidance on how to compare their hospital’s results against the comparative results from the database.

Most hospitals simply compare their percent positive scores against the database averages and do not attempt any statistical comparisons. However, given the large number of hospitals and respondents in the database, and the average number of respondents per hospital (average N = 503), it is likely that even small differences in a hospital’s scores compared to the database will be statistically significant. Therefore, to help hospitals simplify comparisons against the database and provide conservative guidance on what level of difference would be considered meaningful, the report recommends that hospitals use a 5% difference in scores as a rule-of-thumb to determine whether its scores can reasonably be considered higher or lower than the database scores. The following text is provided (on pages 32 of the 2012 Database Report):

Use a 5 percent difference as a rule of thumb when comparing your hospital’s results to the database averages. Your hospital’s percent positive score should be at least 5 percent higher than the database average to be considered “better,” and should be at least 5 percent lower to be considered “lower” than the database average. A 5 percent difference is likely to be statistically significant for most hospitals given the number of responses per hospital, and is also a meaningful difference to consider.

When examining changes in survey scores over time for trending hospitals that submitted data to the database more than once, the average change in percent positive scores across the patient safety culture composites was an increase of 2 percentage points (ranging from 1 to 3 percentage points), which demonstrates that, on average, change on the survey appears to be incremental over time for hospitals, and demonstrates that a 5% difference works as a conservative rule-of-thumb when examining differences in scores.

Appendixes: Overall and trending results by hospital and respondent characteristics. In addition to the main Comparative Database Report, there is an accompanying Appendix volume (Part 2) that presents overall results and trending results by hospital characteristics (bed size, teaching status, ownership, and geographic region) and by respondent characteristics (work area/unit, staff position, and extent of interaction with patients). These appendix statistics are presented for hospitals that want to dive deeper into the data and make more specific comparisons of their hospital’s results against hospitals of similar size, or compare their physicians’ responses to those from the entire database.

2. Information Collection Procedures

Information collection for the AHRQ Hospital Survey on Patient Safety Culture Comparative Database occurs in a periodic data collection cycle every two years from June 1 to June 15. Information collection procedures for submitting and processing data are shown in Figure 1.

Figure 1. Hospital SOPS Comparative Database Data Submission

Step 1: Call for Data Submission. Beginning June 1 every two years, announcements about the opening of data submission go out through various publicity sources. AHRQ’s electronic newsletter targets approximately 50,000 subscribers. In addition, the AHRQ Surveys on Patient Safety Culture listserv targets approximately 22,000 subscribers. An example of an email announcement calling for data submission is shown in Attachment I, Email # 1. In addition, the AHRQ web site has public information about the yearly timeline and instructions for data submission (http://www.ahrq.gov/legacy/qual/hospsurveydb/y2dbsubmission.htm). Through these efforts, U.S. hospitals are made aware of and invited to submit their survey data to the database.

As the administrator of the database and under contract with AHRQ, Westat provides free technical assistance to submitting hospitals and maintains a dedicated email address ([email protected] ) and toll-free phone number (1-888-852-8277).

Step 2: Registration for Potential Participants. A secure data submission web site allows interested parties such as hospitals and health systems to register and submit data. The login page for the web site (https:// lSops-Database.org/hospital) is shown in Attachment J, Figure 1. On the login page, users register by clicking “Register for an account.” Registration is one page that takes about 3 minutes to complete and asks for contact information and other basic information, shown in Attachment G. After registering, if registrants are deemed eligible to submit data, two separate, automated emails are sent to provide them with a username and password and information needed in the next steps of the data submission process (Attachment I, Emails # 2 and # 3).

Once users have a username and password, they can enter the main page menu of the web site (shown in Attachment J, Figure 2). Information about eligibility requirements, data use agreements, and data file specifications regarding how to prepare their data for inclusion in the SOPS database is posted and can be reviewed.

Step 3: Enter Facility Information and Upload Questionnaire. At this step, users provide information about each of their facilities, such as Medicare Provider ID, AHA ID, point-of-contact, methods of survey administration, overall response rate, and--if they do not have an AHA ID--other facility characteristics (bed size, teaching status, and ownership)(Attachment H). They also upload their survey questionnaire that they administered to enable us to determine whether any changes were made to the survey (Attachment J, Figure 3).

Step 4: Submit Data Use Agreement. To protect the confidentiality of all participating hospitals, a duly authorized representative from the hospital must sign a data use agreement (DUA) (Attachment F). The DUA language was reviewed and approved by AHRQ’s general counsel. The DUA states that the hospital’s data will be handled in a secure manner using necessary administrative, technical and physical safeguards to limit access to it and maintain its confidentiality. In addition, the DUA states the data will be used for the purposes of the database, that only aggregated results will be reported, and that the hospital will not be identified by name. Data are not included in the database without this signed data use agreement. Users can fax and/or mail a copy of the signed agreement.

Step 5: Upload Data File(s). At this step, users are asked to upload their individual-level survey data for each hospital (Attachment J, Figure 4). Data submitted through the secure data submission web site are encrypted to ensure secure, confidential transmission of the survey data. Data are accepted in Microsoft Excel® format since this is the format preferred by hospitals. Users must upload one data file per facility. Hospital data files must contain the site’s corresponding Medicare Provider ID and AHA ID (if available). The data file specifications (Attachment K) are provided to data submitters to ensure that users submit standardized and consistent data in the way variables are named, coded, and formatted.

Once a data file is uploaded, a separate load program developed in Visual Basic (VB) reads the submitted files and loads them into the SQL database that stores the data. A data quality report is then produced and made available to the participant. This report displays item frequencies and flags out-of-range values and incorrectly reverse-coded items. If there are no problems with the data, an acknowledgement of data upload will be granted via an automatic email. If data are improperly coded, an automatic email informs the participant of the problem. Users are expected to fix any errors and resubmit their data file(s) for processing. Once there are no problems, an email is sent to the facility contact via the database submission web site indicating their data has received final acceptance.

Step 6: Approve Data Submission. Once all of the information required for submission has been submitted and approved, an email is sent to the facility contact indicating that their data has received final acceptance.

3. Methods to Maximize Response Rates

AHRQ makes a number of toolkit materials available to assist hospitals with the SOPS surveys. The Hospital SOPS has a Survey User’s Guide that gives users guidance and tips about survey administration on the following topics: planning; selecting a sample; determining their data collection method; data collection procedures, with a section on web surveys; and analyzing data and producing reports. The Survey User’s Guide also gives hospitals tips about how to increase response rates through publicity efforts, top management support, use of incentives, and following all steps of proper data collection protocols.

As noted earlier in this document under Information Collection Procedures, Step 1--Call for Data Submission, beginning June1 every two years, announcements about the opening of data submission go out through various publicity sources as a way to boost hospital participation in the database. AHRQ’s patient safety and electronic newsletters target approximately 50,000 subscribers. In addition, the AHRQ Surveys on Patient Safety Culture listserv targets approximately 22,000 subscribers. AHRQ, through its contractor Westat, provides free technical assistance to users through a dedicated email box and toll-free phone number. In addition, reminders are sent to database registrants to remind them of the deadline for data submission.

4. Tests of Procedures

Input and Feedback for the Development of the SOPS Database Submission System. Because the Surveys on Patient Safety Culture are public-use instruments, the SOPS program has generally modeled its data submission process after those utilized by the CAHPS Database that has been in operation for many years. SOPS staff consulted with CAHPS Database staff and programmers to determine best practices for data submission. This information, as well as feedback obtained during the provision of technical assistance each year the database has been running, has been used to improve the SOPS online data submission system and process over time.

5. Statistical Consultants

Joann Sorra, PhD

Westat

1600 Research Blvd.

Rockville, MD 20878

[email protected]

Naomi Dyer, PhD

Westat

1600 Research Blvd.

Rockville, MD 20878

[email protected]

1 An alternative method would be to report a straight percentage of positive response across all respondents, but this method would give greater weight to respondents from larger hospitals since they account for almost twice as many responses as those from smaller hospitals.

14

ICR: Hospital Survey on Patient Safety Culture Comparative Database (201312-0935-001)