Supporting Statement Part A Final 9.09.2014

SUPPORTING STATEMENT

Part A

Updating and Expanding the AHRQ Quality Indicators Toolkit for Hospitals

Version: September 9, 2014

Agency of Healthcare Research and Quality (AHRQ)

Table of contents

A. Justification 3

1. Circumstances that make the collection of information necessary 3

2. Purpose and use of information 6

3. Use of Improved Information Technology 6

4. Efforts to Identify Duplication 7

5. Involvement of Small Entities 7

6. Consequences if Information Collected Less Frequently 7

7. Special Circumstances 7

8. Consultation outside the Agency 7

9. Payments/Gifts to Respondents 8

10. Assurance of Confidentiality 8

11. Questions of a Sensitive Nature 8

12. Estimates of Annualized Burden Hours and Costs 8

13. Estimates of Annualized Respondent Capital and Maintenance Costs 10

14. Estimates of Annualized Cost to the Government 10

15. Changes in Hour Burden 11

16. Time Schedule, Publication and Analysis Plans 11

17. Exemption for Display of Expiration Date 11

List of Attachments 12

A. Justification

1. Circumstances that make the collection of information necessary

The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:

1. research that develops and presents scientific evidence regarding all aspects of health care; and

2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and

3. initiatives to advance private and public efforts to improve health care quality.

Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.

An important part of AHRQ’s mission is to disseminate information and tools that can support improvement in quality and safety in the U.S. health care community. This proposed information collection supports that part of AHRQ's mission by developing and evaluating a toolkit that will enable hospitals to effectively use AHRQ's Quality Indicators (QIs).

AHRQ has developed sets of Quality Indicators (QIs) that can be used by the Agency and others to document quality and safety conditions at U.S. hospitals. Three sets of QIs are particularly relevant for hospitals and include: the Inpatient Quality Indicators (IQIs), the Patient Safety Indicators (PSIs), and the Pediatric Quality Indicators (PDIs). The IQIs contain measures of volume, mortality, and utilization for common medical conditions and major surgical procedures. The PSIs are a set of measures to screen for potentially preventable adverse events that patients may experience during hospitalization. The PDIs measure the quality of pediatric health care, mainly focusing on preventable complications that occur as a consequence of hospitalization among pediatric patients. These QIs have been previously developed and evaluated by AHRQ, and are in use at a number of hospitals throughout the country. The QIs and supportive documentation on how to work with them are posted on AHRQ’s website at www.qualityindicators.ahrq.gov.

Despite the availability of the QIs as tools to help hospitals assess their performance, many U.S. hospitals have limited experience with the use of such measurement tools, or in using quality improvement methods to improve their performance as assessed by these measures. To this end, RAND has previously contracted with AHRQ to develop an AHRQ Quality Indicators Toolkit for Hospitals (Toolkit). This Toolkit is publicly available and is posted on AHRQ’s website at http://www.ahrq.gov/professionals/systems/hospital/qitoolkit/index.html. The Toolkit assists hospitals in both using the QIs and improving the quality and safety of the care they provide, as measured by those indicators. As such, the Toolkit includes: (1) instruction on how a hospital can apply the QIs to its inpatient data to estimate rates for each indicator; (2) methods the hospital can use to evaluate these QI rates for identifying opportunities for improvement; (3) strategies for implementing interventions (or evidence-based best practices); (4) methods to measure progress and performance on the QIs; (5) tools for evaluating the cost-effectiveness of these changes; and (6) discussion of the value of using the QIs for quality improvement as well as potential challenges and barriers to quality improvement efforts that incorporate the QIs and how to help overcome them. OMB approval was obtained for the development and evaluation of the original Toolkit in 2012, Development and Evaluation of AHRQ’s Quality Indicators Improvement Toolkit ( OMB # 0935-0164), which consisted of a protocol very similar to the one described in this statement.

Since the release of the Toolkit in 2012, the QIs have been updated and expanded, best practices have advanced, and many hospitals have improved their understanding of their quality improvement needs as well as increased their familiarity with the use of the Toolkit. These factors all point to the critical need to update the Toolkit. AHRQ has funded RAND which partners with the University HealthSystem Consortium (UHC) to update and expand the Toolkit, and field test the updated Toolkit with hospitals as they carry out initiatives designed to improve performance on the QIs.

This research has the following goals:

1. To assess the usability of the updated Toolkit for hospitals – with an emphasis on the Pediatric Quality Indicators (PDI) - in order to improve the Toolkit, and

2. To examine hospitals’ experiences in implementing interventions to improve their performance on the AHRQ QIs, the results of which will be used to guide successful future applications of the Toolkit.

As shown in Figure 1, an alpha version of the updated and expanded Toolkit will be developed and then be field tested. During the field test, the proposed evaluation described herein will assess the usability of the updated Toolkit for hospitals, and it will examine their experiences in implementing interventions to improve their performance on the AHRQ QIs. Using results from the evaluation, the alpha updated Toolkit will be revised to yield a final updated Toolkit that will be effective in supporting hospitals’ quality improvement efforts.

Figure 1. Logic Model of the AHRQ Quality Indicator Improvement Toolkit Project Update and Expansion

The evaluation is the part of this project that involves primary data collection, which will include data on hospital improvement experiences and on the usability of the Toolkit. In the field test of the alpha Toolkit, as part of a larger group of implementation hospitals, the six (6) case study hospitals will apply the tools it contains in their quality improvement activities aimed at improving their performance on the AHRQ QIs. All of the six case study hospitals will provide some degree of pediatric care, ranging from freestanding children’s hospitals to general hospitals with pediatric units. The hospitals can choose any of the AHRQ QIs, but we will ask that each hospital choose at least one PDI for evaluation. This is because the evaluation focuses on the PDIs as they are the newest part of the Toolkit and have not been previously evaluated. Freestanding children’s hospitals will choose two PDIs for evaluation (as they only serve children), and children’s hospitals within hospitals and general hospitals will choose one PDI and one PSI or IQI for evaluation. Hospitals will use their current values for the QIs to identify priorities for improvement actions, and then will develop and carry out an implementation plan designed to bring about those improvements. In each step of this process, they will make use of tools in the Toolkit designed to support the relevant implementation step.

To achieve the goals of this project the following data collections will be implemented:

1) Pre/post-test interview protocol -- consisting of both open and closed ended questions will be administered prior to implementation of the Toolkit and again post implementation. The purpose of this data collection is to obtain data on the steps the hospitals took to implement actions to improve performance on the QIs; their plans for making process changes; and their experiences in achieving changes and perceptions regarding lessons learned that could be shared with other hospitals. (See Attachment B).

2) Update protocol – consisting of both open and closed ended questions will be administered three times during the study (quarterly during the implementation year). The purpose of this data collection is to capture longitudinal data regarding hospitals’ progress in implementing changes, successes and challenges, and plans for subsequent actions. These data will include descriptive information on changes over time in the hospitals’ implementation actions and how they are using the Toolkit, as well as experiential information on the perceptions of participants regarding the improvement implementation process and its effects. It also ensures the collection of information close to pertinent events, which avoids the recall bias associated with retrospective reporting of experiences. (See Attachment C).

3) Usability testing protocol – also consisting of both open and closed ended questions will be administered once at the end of the evaluation period. The purpose of this data collection is to gather information from the hospitals on how they used each tool in the updated Toolkit, the ease of use of each tool, which tools were most helpful, suggested changes to improve each tool, and suggestions for other tools to add to the updated Toolkit. This information will be used in the revisions of the updated Toolkit following the end of the field test. (See Attachment D).

The proposed data collection to be carried out in evaluating the updated Toolkit is an essential part of the Toolkit revision process because it will gather important information from hospitals, as end users of the product, on what is necessary to ensure that the updated and expanded Toolkit is both useful and readily usable by them. Only by having this direct feedback from the participating hospitals can the Toolkit effectively support hospitals in applying quality improvement efforts related to performance on the AHRQ QIs.

This evaluation is being conducted by AHRQ through its contractor, the RAND Corporation under contract number HHSA290201000017I, pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2).

2. Purpose and Use of Information

All the information obtained from the proposed data collection will be used to strengthen the updated Toolkit before finalizing and disseminating it to hospitals for their use. First, information will be collected from the six hospitals participating in the Toolkit field test about their experiences in implementing performance improvements related to the AHRQ QIs, which will be used to prepare experiential case examples for inclusion in the Toolkit as a resource for other hospitals. Second, feedback will be elicited from them about the usability of the Toolkit, which will be applied to modify and refine the Toolkit so that it is as responsive as possible to the needs and priorities of the hospitals for which it is intended.

3. Use of Improved Information Technology

This collection of information will not involve the use of automated or electronic collection techniques. All of the interviews will be conducted either by telephone or in person during site visits to hospitals.

4. Efforts to Identify Duplication

As stated above, two types of information will be collected in the interviews: information on hospitals’ experience in implementing quality improvements related to the AHRQ QIs and feedback from the hospitals about the usefulness and usability of the Toolkit. Both sets of information will be unique to this project. The information on hospitals’ implementation experience is unique because the experience will relate specifically to their work with the updated AHRQ Toolkit, including the PDIs and (where applicable) PSIs and IQIs as well as related improvement actions. Although it is known from published materials and previous experience that various factors influence implementation success, the instruments used here will collect data on the specific status of these hospitals on these factors, which also is unique. The feedback on the updated and expanded Toolkit is completely unique to this project because it is tailored specifically to improving the products being developed in the project.

5. Involvement of Small Entities

None of the participating hospitals would be considered a small business.

6. Consequences if Information Collected Less Frequently

This data collection activity is a one-time collection done as part of the evaluation for the update and expansion of the Toolkit. If the data collection were not performed, activities in support of AHRQ’s Congressionally mandated activities to improve the quality and safety of health care for the country would be severely hindered, because the evaluation information is essential to revising a Toolkit that effectively supports performance improvements by hospitals on the AHRQ Quality Indicators.

7. Special Circumstances

This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.

8. Federal Register Notice and Outside Consultations

8.a. Federal Register Notice

As required by 5 CFR 1320.8(d), notice was published in the Federal Register on May 12, 2014 for 60 days (see Attachment E). No substantive comments were received.

8.b. Outside Consultations

RAND and its subcontractor, UHC, consulted with representatives of the hospitals that will be participating in the evaluation to obtain their view and suggestions regarding the evaluation data collection process, and will consult with them as the evaluation interviews are scheduled to minimize disruption of their work processes. Further, these data collection methods have been used by RAND in previous studies; they are proven methods that will be applied in this evaluation.

RAND has already consulted with and will continue to consult with Dr. Pamela Owens of AHRQ who oversees the development of the AHRQ QIs themselves. There are no unresolved issues at this time.

9. Payments/Gifts to Respondents

No payments or gifts will be provided to the hospitals participating in the evaluation.

10. Assurance of Confidentiality

Individuals and organizations will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose. The responses will be aggregated with those of other respondents before any information is reported to any other party outside of the research team (see Attachments C, D, F).

Individuals and organizations will be assured of the confidentiality of their replies under Section 944(c) of the Public Health Service Act.  42 U.S.C. 299c-3(c).  That law requires that information collected for research conducted or supported by AHRQ that identifies individuals or establishments be used only for the purpose for which it was supplied.

Information that can directly identify the respondent will be collected. We will collect respondents’ names, organization affiliation (including business address), telephone number and email address. Respondents’ names and organizational affiliations will be contained in the hardcopy and softcopy notes from the interviews conducted by telephone or during the site visits. The interview notes for the group interviews will not associate the identity of individual subjects with specific comments made during the interview. Because the data collected are organizational rather than personal information, these data are private but are not considered sensitive.

11. Questions of a Sensitive Nature

The evaluation does not include any questions that would be considered sensitive.

Verbal informed consent will be obtained from all interview participants before starting each interview (see Attachments C, D, and F), in compliance with the requirements of RAND’s IRB, which reviewed our project and granted an exemption (see Attachment G).

12. Estimates of Annualized Burden Hours and Costs

Exhibit 1 shows the estimated annualized burden hours for the respondents' time to participate in this information collection. Three protocols will be used to collect data from respondents in interviews that will take one hour each. The pre/post-test interview protocol will be administered twice – at the beginning and end of the field-test year. The pre-test interviews will be performed as one-hour group interviews conducted with the six hospitals’ implementation teams at the start of the year. Each implementation team is expected to consist of about 5 people. At the end of the year, post-test interviews will be conducted during site visits with the six hospitals. At each site visit, data will be collected through one-hour interviews with the hospital’s implementation team as well as through other group interviews performed separately with each of the key stakeholder groups – physicians, nurses, clerks, and others (for a total of 5 post-implementation interviews). Again, each implementation team and stakeholder group is expected to consist of about 5 people. The additional data from the stakeholder groups will allow triangulation of variations in perceptions and experiences among different groups, of which the implementation teams might not be aware.

The quarterly update protocol will be administered quarterly to 2 hospital staff members from each hospital during the year (in months 3, 6, and 9). The usability testing protocol will be administered to 4 staff members once at the end of the evaluation period. The total burden is estimated to be 240 hours.

Exhibit 2 shows the estimated annualized cost burden associated with the respondents' time to participate in the evaluation. The total cost burden is estimated to be $7,179.

Exhibit 1.  Estimated annualized burden hours

Data Collection	Number of respondents	Number of responses per hospital	Hours per response	Total Burden hours
Pre/Post-Test Interview Protocol	150 (25 respondents per hospital)	120 respondents with 1 interview each (120 interviews total) 30 respondents with 2 interviews each (60 interviews total)	1	180
Quarterly Update Protocol	12 (2 respondents per hospital)	3	1	36
Usability Testing Protocol	24 (4 respondents per hospital)	1	1	24
Total	186	NA	NA	240

Exhibit 2. Estimated annualized cost burden

Data Collection	Number of respondents	Total Burden hours	Average Hourly Wage Rate*	Total Cost Burden
Pre/Post-Test Interview Protocol	150	180	$29.91	$5,384
Quarterly Update Protocol	12	36	$29.91	$1,077
Usability Testing Protocol	24	24	$29.91	$718
Total	186	240	NA	$7,179

*Based upon the mean of the average wages taken from an average of hourly rates for occupations likely to be involved in the QI process (registered nurses, nurse practitioners, medical records and health information technicians, statisticians, and health technologists and technicians). Statistics are taken from the General Medical and Surgical Hospitals industry category in the May 2012 National Industry-Specific Occupational Employment and Wage Estimates from the Bureau of Labor Statistics, U.S. Department of Labor, accessed on January 22, 2014 [www.bls.gov/oes/]

13. Estimates of Annualized Respondent Capital and Maintenance Costs

Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. There are no direct costs to respondents other than their time to participate in the study.

14. Estimates of Annualized Cost to the Government

Exhibit 3 shows the estimated total and annualized cost of this project to the government. The estimated total cost for the evaluation work is $316,665 over the three-year project, with an annualized cost of $105,554. These costs were developed based on estimates of staff days required, to which administrative expenses are applied, and based on airfare, hotel, and per diem costs for staff travel for the site visits at the end of the evaluation. We have also included the cost of AHRQ oversight of the project

Exhibit 3. Estimated Cost of the Evaluation

Cost Component	Total Cost	Annualized Cost
Protocol Development	$59,619	$19,873
Data Collection Activities	$134,142	$44,714
Data Analysis	$68,562	$22,854
Publication of Results	$35,771	$11,924
Travel for Site Visits	$17,438	$5,812
AHRQ Oversight	$1,133	$378
Total	$316,665	$105,554

Exhibit 4: Annual Cost to AHRQ for IAA Oversight

Tasks/Personnel	Staff Count	Annual Salary	% of Time	Cost
Research Support: GS-14, Step 3 average	1	$113,346	1%	$1,133.46
Grand total				$1,133.46

Annual salaries based on 2014 OPM Pay Schedule for Washington/DC area: http://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2014/DCB.pdf

Note that these oversight costs are included as “AHRQ Oversight” in Exhibit 3.

15. Changes in Hour Burden

This is a new information collection.

16. Time Schedule, Publication and Analysis Plans

Data collected will be analyzed using standard qualitative data analysis methods, by aggregating responses for each topic area and comparing and contrasting the information provided by respondents at the participating hospitals. For analyzing implementation experiences, a focus will be given to key successes, challenges or barriers encountered, progress toward achieving intended improvements, prospects for sustainability, and advice to others pursuing similar improvements. For the feedback on the Toolkit usability, comments and suggestions for each tool will be aggregated and summarized, including identification of conflicting reactions as well as concrete suggestions for improving the tool. The analysis of effects on the QIs will be done by calculating differences in each hospital’s scores for each QI and testing the significance of those differences.

The results of the evaluation will be used to refine and finalize the Toolkit, which then will be disseminated for use by hospitals across the country. In addition, a report will be prepared for AHRQ, presenting the full results of this work, and articles will be prepared and submitted for publication in health-related peer-review and/or social science research journals. Of note, we have timed our implementation and evaluation of the Toolkit to take place 6 months after the introduction of ICD-10 codes, which will take place in October 2014. This will allow hospitals to adjust to the new ICD-10 codes. The table below presents the project’s current schedule:

Task/Activity	Timeline and Proposed Date of Completion
Conduct pre-implementation interviews	February 2015
Hospitals start implementing improvements and using Toolkit	February 2015
Conduct quarterly update interviews	May 2015, August 2015, December 2015
Conduct post-implementation interviews	February 2016
Conduct interviews on tool usability	February 2016
Revise Toolkit based on evaluation results	April 2016
Dissemination of Toolkit	May 2016
Report and journal articles	June 2016

17. Exemption for Display of Expiration Date

AHRQ does not seek this exemption.

List of Attachments:

Attachment A -- Healthcare Research and Quality Act of 1999

Attachment B -- Pre-post interview protocol

Attachment C -- Quarterly update protocol (includes informed consent statement)

Attachment D -- Toolkit usability protocol (includes informed consent statement)

Attachment E -- 60 Day Federal Register Notice

Attachment F -- Informed consent form for pre/post interview

Attachment G -- RAND Human Subjects Protection Committee (IRB) Exemption Notice

Attachment H – Recruitment Letter