Supporting Statement Part B -- HSOPS DB Renewal 10172016

SUPPORTING STATEMENT

Part B

Collection of Information for

Agency for Healthcare Research and Quality’s (AHRQ)

Hospital Survey on Patient Safety Culture Comparative Database

Version February 11, 2016

Agency of Healthcare Research and Quality (AHRQ)

Table of contents

B. Collections of Information Employing Statistical Methods 3

1. Respondent universe and sampling methods 3

2. Information Collection Procedures 6

3. Methods to Maximize Response Rates 8

4. Tests of Procedures 9

5. Statistical Consultants 9

B. Collections of Information Employing Statistical Methods

1. Respondent universe and sampling methods

The AHRQ Hospital Survey on Patient Safety Culture (Hospital SOPS) Comparative Database serves as a central U.S. repository for data from the survey and AHRQ houses the largest database of the survey’s results. However, the database is comprised of data that are voluntarily submitted by hospitals that have administered the survey, and is not a statistically selected sample, nor is it a representative sample of all U.S. hospitals. The Hospital SOPS was developed and pilot tested with OMB approval (OMB NO. 0935-0115; Approved 2/4/2003). The original Hospital SOPS Comparative Database was last approved on September 26, 2013 (OMB NO. 0935-0162). Comparative results from 653 hospitals that participated in the 2014 database are publicly available on the AHRQ Web site at: http://www.ahrq.gov/professionals/quality-patient-safety/patientsafetyculture/hospital/hosp-reports.html.

An AHRQ Hospital Survey on Patient Safety Culture Comparative Database Report has been produced yearly since 2007 until 2012 and then every two years beginning in 2012. The three most recent reports (2011, 2012, and 2014) are on the AHRQ web site at http://www.ahrq.gov/professionals/quality-patient-safety/patientsafetyculture/hospital/hosp-reports.html.

Table B-1 presents the 653 hospitals in the 2014 Comparative Database categorized by whether they had only submitted once or more than once.

Table B-1. Statistics for Nontrending and Trending Hospitals in 2014 Database

Overall Statistic	Nontrending (Submitted Once)	Trending (Submitted More Than Once)	Total 2012 Database
Number of hospitals	294	359	653
Number of individual survey respondents	151,573	253,708	405,281

The number of hospitals in the U.S. is estimated to be 5,795 (U.S. Census Bureau: http://www.census.gov/library/publications/2011/compendia/statab/131ed/health-nutrition.html, Table 172 Hospitals; accessed January 19, 2016.) However, the latest AHRQ Hospital Survey on Patient Safety Culture Comparative Database Report consists of data from 653 hospitals which represents only 11.3% of the total estimated population of U.S. hospitals.

In an effort to discern the comparability of the 653 hospitals in the 2014 Database Report to hospitals in the population, AHRQ presents statistics comparing the characteristics of the 653 hospitals (bed size, teaching status, ownership & control, region) against characteristics of U.S. hospitals based on those registered with the American Hospital Association (AHA). Although the AHA data set, which is updated every 2 years, does not capture characteristics on the entire population of U.S. hospitals, it is used as the comparison because it is the best source of data on hospital characteristics and includes more hospitals than any other data set.

Comparisons of the 653 database hospitals against characteristics obtained from the AHA (the 2011 AHA Annual Survey of Hospitals Database © Health Forum, LLC) are provided in Tables B-2 to B-5 and are displayed in Chapter 3 of the 2014 Comparative Database Report, available at: http://www.ahrq.gov/professionals/quality-patient-safety/patientsafetyculture/hospital/hosp-reports.html.

The tables show that the 653 database hospitals are similar to the distributions of characteristics from AHA-registered hospitals. However, the database over-represents larger hospitals, teaching hospitals, and nongovernment hospitals. In addition, the database distribution under-represents the West North Central and West South Central hospitals and over-represents the Mid Atlantic, South Atlantic/Associated territories, and East North Central hospitals compared to the distribution of AHA-registered U.S. hospitals.

Table B-2. Distribution of Database Hospitals and Respondents by Bed Size

(Compared to AHA-registered U.S. Hospitals)

Bed Size	AHA-registered U.S. Hospitals		2014 Database Hospitals		2014 Database Respondents
	Number	Percent	Number	Percent	Number	Percent
6-24 beds	701	11%	37	6%	3,582	1%
25-49 beds	1,451	23%	96	15%	15,168	4%
50-99 beds	1,276	20%	112	17%	31,911	8%
100-199 beds	1,280	20%	148	23%	64,451	16%
200-299 beds	684	11%	98	15%	74,418	18%
300-399 beds	409	6%	63	10%	60,370	15%
400-499 beds	201	3%	35	5%	44,001	11%
500 or more beds	315	5%	64	10%	111,380	27%
TOTAL	6,317	100%	653	100%	405,281	100%

Table B-3. Distribution of Database Hospitals and Respondents by Teaching Status

(Compared to AHA-registered U.S. Hospitals)

Teaching Status	AHA-registered U.S. Hospitals		2014 Database Hospitals		2014 Database Respondents
	Number	Percent	Number	Percent	Number	Percent
Teaching	1,537	24%	243	37%	244,102	60%
Non-Teaching	4,780	76%	410	63%	161,179	40%
TOTAL	6,317	100%	653	100%	405,281	100%

Table B-4. Distribution of Database Hospitals and Respondents by Ownership and Control

(Compared to AHA-registered U.S. Hospitals)

Ownership and Control	AHA-registered U.S. Hospitals		2014 Database Hospitals		2014 Database Respondents
	Number	Percent	Number	Percent	Number	Percent
Government (Federal or non-Federal)	1,554	25%	140	21%	75,428	19%
Non-Government (voluntary/non-profit or proprietary/investor owned)	4,763	75%	513	79%	329,853	81%
TOTAL	6,317	100%	653	100%	405,281	100%

Table B-5. Distribution of Database Hospitals and Respondents by Geographic Region

(Compared to AHA-registered U.S. Hospitals)

Region	AHA-registered U.S. Hospitals			2014 Database Hospitals		2014 Database Respondents
	Number	Percent		Number	Percent	Number	Percent
New England	261	4%		15	2%	15,479	4%
Mid Atlantic	573	9%		79	12%	75,378	19%
South Atlantic/Associated Territories	1,009	16%		134	21%	86,810	21%
East North Central	916	15%		162	25%	111,984	28%
East South Central	521	8%		55	8%	22,861	6%
West North Central	799	13%		48	7%	15,845	4%
West South Central	1,079	17%		70	11%	24,685	6%
Mountain	508	8%		17	3%	10,943	3%
Pacific/Associated Territories	651	10%		73	11%	41,296	10%
TOTAL	6,317	100%		653	100%	405,281	100%
NOTE: States and territories are categorized into AHA-defined regions as follows:
New England: ME, NH, VT, MA, RI, CT			West North Central: MN, IA, MO, ND, SD, NE, KS
Mid Atlantic: NJ, NY, PA			West South Central: AR, LA, OK, TX
South Atlantic/Associated territories: DE, MD, DC, VA, WV, NC, SC, GA, FL, Puerto Rico, Virgin Islands			Mountain/Pacific/Associated Territories: AZ, AK, CA, CO, HI, ID, MT, NM, NV, OR, UT, WA, WY, American Samoa, Guam, Marshall Islands, Northern Mariana Islands
East North Central: OH, IN, IL, MI, WI
East South Central: KY, TN, AL, MS

Trending results are also presented in the Comparative Database Report and are based only on the subset of hospitals that have submitted data more than once. Trending results display the average percentage of positive response for previous and most recent administrations of the survey, the change in scores (as a difference between the average scores), the maximum increase and decrease in scores, and the average increase and decrease in percent positive scores over time.

Hospitals that submit data to the database receive a free, customized feedback report that displays the hospital’s results against the database. Hospitals that do not submit data to the database can still compare their results using the Comparative Database report. The report provides instructions on how to calculate percent positive scores and compare them against the database. As part of a toolkit of support materials for the Hospital SOPS survey, hospitals can use a Microsoft® Excel-based Data Entry and Analysis Tool that is an Excel file with macros that will automatically produce graphs and charts of a hospital’s results once data are entered into a data sheet. Many hospitals use this tool to produce their results.

In the overall database report, hospitals are provided with a detailed description and explanation of the statistics that are presented and given examples and guidance on how to compare their hospital’s results against the comparative results from the database.

Most hospitals simply compare their percent positive scores against the database averages and do not attempt any statistical comparisons. To help hospitals simplify comparisons against the database and provide conservative guidance on what level of difference would be considered meaningful, the report recommends that hospitals use a 5 percentage point difference in scores as a rule-of-thumb to determine whether its scores can reasonably be considered higher or lower than the database scores.

2. Information Collection Procedures

Information collection for the AHRQ Hospital Survey on Patient Safety Culture Comparative Database occurs in a periodic data collection cycle every two years from June 1 to June 15. The next submission period is June 2017. Information collection procedures for submitting and processing data are shown in Figure 1.

Figure 1. Hospital SOPS Comparative Database Data Submission

Step 1: Call for Data Submission. Announcements about the opening of data submission go out through various publicity sources. AHRQ’s patient safety and electronic newsletters target approximately 50,000 subscribers. In addition, the AHRQ Surveys on Patient Safety Culture listserv targets approximately 27,000 subscribers. An example of email announcements calling for data submission is shown in Attachment D, Email # 1 and # 3. Through these efforts, U.S. hospitals are made aware of and invited to submit their survey data to the database.

As the administrator of the database and under contract with AHRQ, Westat provides free technical assistance to submitting hospitals through a dedicated email address ([email protected]) and toll-free phone number (1-888-324-9790).

Step 2: Registration for Potential Participants. A secure data submission Web site allows interested parties such as hospitals and health systems to register and submit data. Registration takes about 3 minutes to complete and asks for contact information and other basic information (see Attachment A). After registering, if registrants are deemed eligible to submit data, an automated email is sent to authenticate the account and update the user (see Attachment D, Email # 2).

Once users are registered and have a password, they can enter the main page menu of the Web site. Information about eligibility requirements, data use agreements, and data file specifications regarding how to prepare their data for inclusion in the SOPS database is posted and can be reviewed.

Step 3: Enter Facility Information and Upload Questionnaire. At this step, POCs provide information about each of their hospitals, such as Medicare Provider ID, AHA ID, point-of-contact, methods of survey administration, overall response rate, and--if they do not have an AHA ID--other facility characteristics (bed size, teaching status, and ownership) (see Attachment C). They also upload their survey questionnaire that they administered to enable us to determine whether any changes were made to the survey (see Attachment G, Figure 1).

Step 4: Submit Data Use Agreement (DUA). To protect the confidentiality of all participating hospitals, a duly authorized representative from the hospital must sign a Data Use Agreement (DUA) (see Attachment B). The DUA language was reviewed and approved by AHRQ’s general counsel. The DUA states that the hospital’s data will be handled in a secure manner using necessary administrative, technical and physical safeguards to limit access to it and maintain its confidentiality. In addition, the DUA states the data will be used for the purposes of the database, that only aggregated results are reported, and that the hospital will not be identified by name. Data are not included in the database without this signed data use agreement. Users can fax and/or mail a copy of the signed agreement.

Step 5: Upload Data File(s). At this step, users are asked to upload their individual-level survey data for each hospital (Attachment G, Figure 2). Data submitted through the secure data submission Web site are encrypted to ensure secure, confidential transmission of the survey data. Data are accepted in Microsoft Excel® format since this is the format preferred by hospitals. Users must upload one data file per hospital. A POC may submit data on behalf of more than one hospital such as in the instance of a hospital system or a survey vendor. On average, a POC submits data on behalf of three hospitals. If a user has multiple hospitals within a health system, users can upload one data file that identifies all of the hospitals in their system. The data file specifications (see Attachment E) are provided to data submitters to ensure that users submit standardized and consistent data in the way variables are named, coded, and formatted.

Once a data file is uploaded, a separate load program developed in Visual Basic (VB) reads the submitted files and loads them into the SQL database that stores the data. A data quality report is then produced and made available to the participant. This report displays item frequencies and flags out-of-range values and incorrectly reverse-coded items. If there are no problems with the data, an acknowledgement of data upload and acceptance will be granted during the user session. If data are improperly coded, the user is informed that the data file failed during the user session by having a message post on the screen. Users are expected to fix any errors and resubmit their data file(s) for processing. Once there are no problems, the user is informed of the acceptance of data during the user session with an online message of acceptance.

Step 6: Approve Data Submission. Once all of the information required for submission has been submitted and approved, an email is sent to the hospital contact indicating that their data has received final acceptance.

3. Methods to Maximize Response Rates

AHRQ makes a number of toolkit materials available to assist hospitals with the SOPS surveys. The Hospital SOPS has a Survey User’s Guide that gives users guidance and tips about survey administration on the following topics: planning; selecting a sample; determining their data collection method; data collection procedures (including a section on Web surveys); and analyzing data and producing reports (at http://www.ahrq.gov/professionals/quality-patient-safety/patientsafetyculture/hospital/index.html). The Survey User’s Guide also gives hospitals tips about how to increase response rates through publicity efforts, top management support, use of incentives, and following all steps of proper data collection protocols. Of the hospitals that voluntarily submitted their data for the 2014 Hospital Survey on Patient Safety Culture Comparative Database, the average response rate was 54% across 653 hospitals.

The Surveys on Patient Safety Culture User Network promotes the database in a number of ways:

1. The GovDelivery listserv of subscribers to the AHRQ Hospital Survey on Patient Safety Culture;

2. National partners that have national reach to hospitals;

3. Users that have contacted the Surveys on Patient Safety Culture technical assistance helpline about the hospital survey;

4. Other outlets such as Webcasts and conferences

As noted earlier in this document under Information Collection Procedures, Step 1 – Call for Data Submission, announcements about the opening of data submission go out through various publicity sources as a way to boost hospital participation in the database. AHRQ’s electronic newsletter targets approximately 50,000 subscribers. In addition, the AHRQ Surveys on Patient Safety Culture listserv targets approximately 27,000 subscribers. AHRQ, through its contractor Westat, provides free technical assistance to users through a dedicated email box and toll-free phone number. In addition, reminders are sent to database registrants to remind them of the deadline for data submission.

4. Tests of Procedures

Input and Feedback for the Development of the SOPS Database Submission System. Because the Surveys on Patient Safety Culture are public-use instruments, the SOPS program has generally modeled its data submission processes after those utilized by the Consumer Assessment of Healthcare Providers and Systems (CAHPS) Database that has been in operation for many years. SOPS staff consulted with CAHPS Database staff and programmers to determine best practices for data submission. This information, as well as feedback obtained during the provision of technical assistance each year the database has been running, has been used to improve the SOPS online data submission system and process over time.

5. Statistical Consultants

Joann Sorra, PhD

Westat

1600 Research Blvd.

Rockville, MD 20878

[email protected]

Naomi Yount, PhD

Westat

1600 Research Blvd.

Rockville, MD 20878

[email protected]

9