Supporting Statement Part A -- Pretest of the Ambulatory Surgery Center SOPS

The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:

1. research that develops and presents scientific evidence regarding all aspects of health care; and

2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and

3. initiatives to advance private and public efforts to improve health care quality.

Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.

Furthermore, AHRQ shall conduct and support research and build private-public partnerships to identify the causes of preventable health care errors and patient injury in health care delivery; develop, demonstrate, and evaluate strategies for reducing errors and improving patient safety; and disseminate such effective strategies throughout the health care industry.

One setting which has demonstrated tremendous growth both in the volume and complexity of procedures being performed is ambulatory surgical and procedure centers (ASCs). ASCs are defined by the Centers for Medicare & Medicaid Services (CMS) as distinct entities that operate exclusively to provide surgical services to patients who do not require hospitalization and are not expected to need to stay in a surgical facility longer than 24 hours (42 C.F.R. §416.2). Many of the services performed in these facilities extend beyond procedures traditionally thought of as surgery, including endoscopy, and injections to treat chronic pain. Currently, there are over 5,300 Medicare-certified ASCs in the U.S., which represents a greater than 54% increase since 2001. In 2007, Medicare paid for more than 6 million surgeries performed in these facilities at a cost of nearly $3 billion. Recent CMS audits suggest infection control deficiencies in these facilities are widespread. For example, preliminary data from 2011 found that 51 percent of ASCs surveyed had an infection control deficiency; 11 percent were considered very serious deficiencies.¹ These findings are only slightly lower than 2010 audits and a 2008 sample of ASCs in three states. In addition, there is anecdotal evidence from states’ ambulatory surgery surveillance systems that preventable events such as medication errors and wrong site, side, patient, procedure, and wrong implant events occur far too frequently.²^,³

Given the widespread impact of ASCs on patient safety, the new Ambulatory Surgery/Procedure Survey on Patient Safety Culture (Ambulatory Surgery SOPS) will measure ASC staff perceptions about what is important in their organization and what attitudes and behaviors related to patient safety culture are supported, rewarded, and expected. The survey will help ASCs to identify and discuss strengths and weaknesses of patient safety culture within their individual facilities. They can then use that knowledge to develop appropriate action plans to improve their practices and their culture of patient safety. This survey is designed for use in ASCs that practice all types of surgical procedures including those that require incisions and less invasive or non-surgical procedures such as gastrointestinal procedures or pain management injections.

This research has the following goals:

1) Develop, cognitively test and modify as necessary the Ambulatory Surgery/Procedure Survey on Patient Safety Culture Questionnaire (Ambulatory Surgery SOPS); and

2) Pretest and modify the questionnaire as necessary; and

3) Make the final questionnaire publically available.

To achieve these goals the following activities and data collections will be implemented:

1) Cognitive interviews. One round of cognitive interviews on the Ambulatory Surgery SOPS will be conducted by telephone with 15 respondents from ASCs. The purpose of these interviews is to understand the cognitive processes the respondent engages in when answering a question on the survey and to refine the survey’s items and composites. These interviews will be conducted with a mix of physicians, management, nurses, surgical technicians, and administrative staff throughout the U.S. from ASCs with varying characteristics (e.g., size, geographic location, and type of ownership). The cognitive interview guide is included in Attachment A.

2) Pretest for the Ambulatory Surgery SOPS. The draft questionnaire will be pretested with physicians and staff from 40 ASCs (see Attachments B). The purpose of the pretest is to collect data for an assessment of the reliability and construct validity of the survey items and composites, allowing for their further refinement. A site-level point of contact (POC) will be recruited in each ASC to manage the data collection at that organization (compile sample information, distribute surveys, promote survey response, etc.). Instructions for the site-level POCs are included in Attachment C. Exhibit 1 includes a burden estimate for the POC’s time to manage the data collection. The draft pretest survey cover letter and follow-up reminder notices are included in Attachments D.

3) Dissemination activities. The final questionnaire will be made publically available through the AHRQ website. This activity does not impose a burden on the public and is therefore not included in the burden estimates in Section 12.

This study is being conducted by AHRQ through its contractor, Health Research & Educational Trust (HRET), and subcontractor, Westat, pursuant to AHRQ’s statutory authority to conduct and support research on health care and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of health care services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2).

2. Purpose and use of Information

The information collected will be used to test and improve the draft survey items in the Ambulatory Surgery SOPS described in Section #1. Psychometric analysis will be conducted on the pretest data to examine item nonresponse, item response variability, factor structure, reliability, and construct validity of the items included in the survey. Because the survey items are being developed to measure specific aspects of patient safety culture in the ambulatory surgery setting, the factor structure of the survey items will be evaluated through multilevel confirmatory factor analysis. On the basis of the data analyses, items or factors may be dropped.

The final survey instrument will be made available to the public for use in ASCs to assess their safety culture from the perspectives of their staff. The survey can be used by ASCs to identify areas for patient safety culture improvement. Researchers are also likely to use the survey to assess the impact of ASC’s patient safety culture improvement initiatives such as the implementation of a surgical safety checklist. This survey is an expansion of AHRQ’s suite of surveys on patient safety culture, which are available on the AHRQ Web site at (http://www.ahrq.gov/professionals/quality-patient-safety/surveys/index.html).⁴ Those surveys have been used by thousands of hospitals, nursing homes, medical offices, and pharmacies across the U.S. to assess patient safety culture. The Ambulatory Surgery SOPS contains new and revised questions and composites that more accurately apply to the ambulatory surgery setting.

3. Use of Improved Information Technology

Data collection will not involve the use of any information technology. This data collection effort is a pretest designed to gather information to examine the psychometric properties (internal consistency reliability, response variability, etc.) of a paper-and-pencil ambulatory surgery safety culture survey instrument. The use of information technology is not needed for this study and is not likely to result in a reduction of burden at this time. The pretest is designed to use primarily a paper survey mode because staff in ASCs do not always have access to the Internet or a computer.

4. Efforts to Avoid Duplication

AHRQ has consulted ambulatory surgery experts and conducted a review of the literature, searching for staff surveys that measure patient safety culture in ambulatory surgery or procedure centers (ASCs). Through these efforts, nine surveys were identified that measure aspects of patient safety culture in ASCs, or health care settings that are similar to ASCs. Descriptive information about a few of the most relevant surveys are provided below.

In 2004, Alvarado et al. administered a brief safety climate survey specifically designed for ASCs that focused primarily on management commitment, general patient safety climate, employee commitment, and change in patient safety.⁵ Dimensions in the survey include disruptive behavior, teamwork, communication openness, response to error, feedback and communication about error, work pressure and pace, and staff training. However, the questions on patient safety climate survey are based on existing established inpatient questionnaires and do not account for the uniqueness of the ambulatory surgery setting.

More recently, the South Carolina Hospital Association (SCHA) and the Harvard School of Public Health (HSPH) have partnered in a project focused on implementation of a surgical safety checklist.⁶ As part of this project, they administered a surgical safety culture survey in hospitals and ASCs to assess surgical team members’ perspectives on factors affecting patient safety in the operating room. The survey items address the following topics: activities supporting checklist implementation; current checklist performance; we speak up, ask for help, discuss mistakes; we share information, listen to each other, and use briefings; we work well together as a surgical team; our physicians are good leaders; people feel valued and respected as part of the surgical team; we routinely perform all elements of the checklist; and perceptions of safety.⁷^,⁸ This survey is designed specifically for clinical staff and focuses on measures likely to be affected by the implementation of a surgical safety checklist. It does not, however, assess patient safety culture from the perspectives of all staff within an ASC and does not assess broader areas of patient safety culture that may not be directly affected by surgical checklist implementation.

5. Involvement of Small Businesses

Some of the ASCs participating in this pretest may be small businesses. The data collection instruments and procedures are designed to minimize burden on individual ASC staff respondents and will not have a significant impact on them.

6. Consequences if Information Collected Less Frequently

This effort is a one-time data collection.

7. Special Circumstances

This request is consistent with the general information guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.

8. Federal Register Notice and Outside Consultations

8.a. Federal Register Notice

As required by 5 CFR 1320.8(d), notice was published in the Federal Register July 8, 2013 for 60 days, and again on September 25, 2013 for 30 days (see Attachment F). No comments were received.

8.b. Outside Consultations

A number of ambulatory surgery experts and consultants were contacted for background interviews on the need for a patient safety culture survey for the ambulatory surgery setting. They will also be consulted for review and comment on the draft survey domains and items—see Attachment G for a list of those consulted both within and outside the Agency thus far. Most of these individuals comprise a formal Technical Expert Panel (TEP) and will provide periodic input throughout the project.

9. Payments/Gifts to Respondents

Cognitive Interview Respondents

To successfully recruit 15 cognitive interview participants for one round of cognitive interview testing, it is appropriate to offer a cash incentive. AHRQ proposes a $200 cash remuneration for ASC physicians and $100 cash remuneration for ASC non-physician staff. Given that the average hourly wage of outpatient care surgeons is greater than $100 per hour and nearly $35 per hour for registered nurses,⁹ the proposed remunerations are intended to partially compensate them for their time.

Pretest Respondents

AHRQ also plans to recruit a site-level POC in each ASC who will manage data collection at that facility (compile sample information, distribute surveys, promote survey response, etc.). AHRQ is not proposing remuneration; rather, we think the benefit of receiving a complimentary feedback report on their survey results is sufficient incentive for participation.

10. Assurance of Confidentiality

Individuals and organizations will be assured of the confidentiality of their replies under Section 944(c) of the Public Health Service Act. 42 U.S.C. 299c-3(c). That law requires that information collected for research conducted or supported by AHRQ that identifies individuals or establishments be used only for the purpose for which it was supplied.

Information that can directly identify the respondent, such as name and/or social security number will not be collected.

11. Questions of a Sensitive Nature

AHRQ believes there are no questions of a particularly sensitive nature included in the survey, but if during cognitive testing sensitivities are discovered, such questions will be modified to ensure they are not of a sensitive nature.

12. Estimates of Annualized Burden Hours and Costs

Exhibit 1 shows the estimated annualized burden hours for the respondents’ time to participate in this research. Cognitive interviews will be conducted with 15 ASC staff (approximately three physicians, six nurses, two medical technicians, two administrative managers, and two administrative assistants) and will take about one hour and 30 minutes to complete. The Ambulatory Surgery SOPS will be completed by 529 ASC staff from 40 facilities (about 13 per facility). Each survey will require approximately 15 minutes to complete. A site-level POC will spend approximately 6 hours administering the Ambulatory Surgery SOPS. The total burden is estimated to be 395 hours annually.

Exhibit 2 shows the estimated annualized cost burden associated with the respondents’ time to participate in this research. The total cost burden is estimated to be $16,173 annually.

Exhibit 1. Estimated annualized burden hours

Form Name	Number of respondents	Number of responses per respondent	Hours per response	Total burden hours
Cognitive interviews	15	1	1.5	23
Pretest for the Ambulatory Surgery SOPS	529	1	15/60	132
POC Administration of the survey	40	1	6	240
Total	584	na	na	395

Exhibit 2. Estimated annualized cost burden

Form Name	Number of respondents	Total burden hours	Average hourly wage rate*	Total cost burden
Cognitive interviews	15	23	$46.52^a	$1,070
Pretest for the Ambulatory Surgery SOPS	529	132	$46.04^b	$6,077
POC Administration of the survey	40	240	$37.61^c	$9,026
Total	584	395	na	$16,173

^a Based on the weighted average wages for 1 Anesthesiologist (29-1061, $108.35), 2 Surgeons (29-1067, $106.48), 2 Administrative Services Managers (11-3011, $37.61), 6 Registered Nurses (29-1141, $34.23), 2 Medical and Clinical Laboratory Technicians (29-2030, $28.90), 1 Licensed Practical or Licensed Vocational Nurse (29-2061, $21.17), and 1 Office and Administrative Support Workers, All Other (43-9199, $16.92).

^b Based on the weighted average wages for 150 Registered Nurses, 85 Office and Administrative Support Workers, 85 Medical and Clinical Laboratory Technicians, 70 Surgeons, 50 Licensed Practical/Vocational Nurses, 49 Anesthesiologists, and 40 Administrative Services Managers.

^c Based on the on the average wages for 1 Administrative Services Managers.

* National Occupational Employment and Wage Estimates in the United States, May 2012, “U.S. Department of Labor, Bureau of Labor Statistics” (available at http://www.bls.gov/oes/current/naics4_621400.htm [for outpatient care setting]

13. Estimates of Annualized Respondent Capital and Maintenance Costs

Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. The only cost to the respondent will be that associated with their time to respond to the information collection, as shown in Exhibit 2.

14. Estimates of Annualized Cost to the Government

Exhibit 3 shows the estimated total and annualized cost to the government for this data collection. Since data collection will last for a year at most, the annualized and total costs are the same. The total cost for the survey is approximately is $431,000.

Exhibit 3. Estimated Total and Annualized Cost

Cost Component	Total Cost	Annualized Cost
Survey Development	$142,600	$142,600
Data Collection Activities	$55,800	$55,800
Data Collection Analysis	$55,800	$55,800
Publication of Results	$5,580	$5,580
Project Management	$7,220	$7,220
Overhead	$164,000	$164,000
Total	$431,000	$431,000

15. Change in Burden

This is a new information collection request.

16. Time Schedule, Publication and Analysis Plans

As soon as OMB approval is received, data collection activities will begin. The estimated time schedule to conduct these activities is shown below:

One round of cognitive interviews (2 months)
Pretest data collection (4 months)
Data analysis, feedback report production, and development of technical report (4.5 months)
Final survey and development of toolkit materials (2 months)

The final version of the Ambulatory Surgery SOPS will be made available in the public domain on the AHRQ Web site.

The goal of AHRQ’s data analysis will be to assess the multi-level psychometric properties of the Ambulatory Surgery SOPS using a multi-staged approach to assess construct validity through factor analyses and intercorrelations among composites and to assess reliability. Psychometric analysis will be conducted on the pretest data to examine item nonresponse, item response variability, factor structure, reliability, and construct validity of the items included in the survey. Because the survey items are being developed to measure specific aspects of patient safety culture in the ambulatory surgery setting, the factor structure of the survey items will be evaluated through multilevel confirmatory factor analysis. On the basis of the data analyses, items or factors may be dropped.

Descriptive Statistics

The means, standard deviations, and response frequencies for the survey items will be examined to ensure that respondents and ASCs exhibit adequate response variability on the survey items. In addition, items will be examined to ensure that there are low rates of missing data (lower than 8% missing response per item). Poorly functioning items will be identified.

Individual Level Factor Analysis

An individual level factor analysis will be conducted to initially examine whether groups of items intended to measure a specific composite are interrelated, ignoring the nesting of respondent data within ASCs. Factor loadings for each item in an a priori composite will be considered as having an adequate contribution to a particular composite or factor if the strength of the item's relationship to that factor (i.e., its factor loading), is 0.40 or greater. Potential composites or factors that the survey items may measure include:

communication;
compliance with procedures;
disruptive behavior;
feedback and communication about error;
leadership support for patient safety;
organizational learning—continuous improvement;
physical environment;
reporting mistakes;
response to mistakes;
staff training and skills;
teamwork; and,
work pressure and pace.

The percent of variance accounted for by the factor will also be examined. The more variance that is accounted for by a factor, the more justifiable it is to combine the items into a single composite score. At least 50% of the variance should be accounted for by the items in a composite.

Intraclass Correlations (ICCs) and Design Effects

Intraclass correlations (ICCs) will be computed for each composite. ICC's determine if substantial variation exists between groups compared to variation within groups. ICCs above 0.05 or 5% indicate that the between group variance is greater than expected by chance and imply that nesting in groups does have an effect on the responses of individuals.

Given that ICCs are likely to be inflated when there are many groups with few individuals within the groups (compared to few groups with many individuals within the groups), we will also examine design effects, which take into account within-group sample size. A design effect of 2 or more implies that group membership or nesting of individuals within groups does have an effect on the responses of the individuals and therefore multilevel modeling should be conducted to account for the multilevel nature of the data.

Multilevel Confirmatory Factor Analysis (MCFA)

Individuals responding to the survey are located within ASCs. When data are nested in groups like this, results from an individual-level factor analysis may be biased or incorrect. Therefore, multilevel confirmatory factor analysis will be conducted on the survey’s a priori composites to examine the structure of the factors at the ASC level of analysis, taking into consideration that the data are nested.

An MCFA will be conducted to test the fit of the measurement model for the survey’s composites, taking into consideration the nested nature of the data at the ASCs level of analysis. We will first evaluate the MCFA results by examining the item factor loadings on the composites. Factor loadings should be 0.40 or greater.

We will also examine overall model fit indices using standard fit statistics: the chi-square, comparative fit index (CFI), and the standardized root mean square residual (SRMR). For chi-square statistics, lower and non-significant chi-squares indicate good fit. The factor structure is determined to adequately fit the data if the CFI is at least 0.90. A value of zero for the SRMR indicates perfect fit, but a value less than 0.08 is considered a good fit.

Reliability Analysis

Reliability analyses will then be performed on the composites to examine whether individuals responded consistently to the items within each composite. Internal consistency reliability will be calculated using Cronbach's alpha. The minimum criterion for acceptable reliability is an alpha of at least 0.70.

Intercorrelations

Intercorrelations among the survey’s composites will also be examined. Intercorrelations will be explored at the individual and ASC level of analysis. While the composites should be correlated since they measure aspects of patient safety culture, the intercorrelations should not be extremely high (0.80 or higher) because very high intercorrelations indicate that the composites may not be unique enough to be considered separate constructs or measures. While there is no steadfast criterion about the magnitude of dimension intercorrelations and construct validity, in general, such correlations should be less than 0.80 for the composites to be considered unique and avoid problems with multicollinearity.

The above analyses will be used to determine which items and composites are poorer functioning and remove them from the survey to derive a final set of items and composites with good psychometric properties and reduce the overall length of the final survey. The Technical Expert Panel will be informed of the data analysis results and be allowed to weigh in on which items to retain or drop when the psychometric results do not provide enough guidance and decisions can be made based on the content value of the items alone.

The final survey will be made available in the public domain for use ASCs and researchers to assess patient safety culture in ambulatory surgery settings.

17. Exemption for Display of Expiration Date

No exemption is being requested.

List of Attachments

Attachment A – Cognitive Interview Guide

Attachment B – Ambulatory Surgery SOPS

Attachment C – Site-level POC Instructions for Survey

Attachment D – Pretest Cover Letter and Reminder Notice for Survey

Attachment E – Federal Register Notice

Attachment F – List of Expert Reviewers and Consultants

1 U.S. Department of Health and Human Services. National Action Plan to Prevent Healthcare-Associated Infections: Roadmap to Elimination— Ambulatory Surgical Centers. April 2012 Draft. Accessed March 1, 2013, from http://www.hhs.gov/ash/initiatives/hai/ ambulatory_surgical_centers.html

2 Pennsylvania Patient Safety Authority. 2011 Annual Report. Accessed March 4, 2013, from http://patientsafetyauthority.org/PatientSafetyAuthority/Pages/AnnualReports.aspx

3 Indiana Department of Health. Indiana Medical Error Reporting System, Final Report for 2011. Accessed March 1, 2013, from http://www.state.in.us/isdh/files/2011_MERS_Report.pdf

4 The Hospital Survey on Patient Safety Culture was pretested (OMB # 0935-0115, exp 1/31/2004) and made available to the public in November 2004; the Nursing Home Survey on Patient Safety Culture was pretested (OMB # 0935-0132, exp. 7/31/2010) and made available to the public in November 2008; the Medical Office Survey on Patient Safety Culture was pretested (OMB # 0935-0131, exp. 7/31/2010) and made available to the public in December 2008; and the Pharmacy Survey on Patient Safety Culture was pretested (OMB # 0935-0183, exp. 8/31/2014) and made available to the public in August 2012.

5 Alvarado CJ, Carayon P, Hundt AS. Patient safety climate (PSC) in outpatient surgery centers. Proceedings of the Human Factors and Ergonomics Society Annual Meeting; 48(14); 2004. p. 1629-33.

6 Singer SJ. (2012b). Surgical Safety Culture Survey. Retrieved Jan 24, 2013, from http://www.safesurgery2015.org/south-carolina-monitoring-program.html

7 Singer SJ (2012). Results from the 2012 Surgical Safety Culture Survey, compared to initial 2011 results [PowerPoint slides].

8 Singer SJ, Kiang M, Huang L, Jiang W. Surgical intervention climate in South Carolina hospitals: Measurement as opportunity to motivate, direct, and assess improvement. The Academy of Management; 2012; Boston, MA.

9 Surgeons (29-1067, $106.48), Registered Nurses (29-1141, $34.23).

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Scott Smith
File Modified	0000-00-00
File Created	2021-01-28

Supporting Statement Part A -- Pretest of the Ambulatory Surgery Center SOPS