Supporting Statement Part B -- National Study of the Hospital AERS 3-26-2010

SUPPORTING STATEMENT

Part B

National Hospital Adverse Event Reporting System:

Questionnaire Redesign and Testing

Version: March 23^rd, 2010

Agency for Healthcare Research and Quality (AHRQ)

Table of contents

B. Revision and Testing of Adverse Event Reporting Survey 3

1. Background 3

2. Survey Revision Process 4

3. Methods to Maximize Response Rates 7

4. Tests of Procedures 7

5. Statistical Consultants 7

B. Collections of Information Employing Statistical Methods

AHRQ proposes to revise and cognitively test the Adverse Event Reporting Survey to reflect changes in the hospital adverse event reporting environment in response to the Patient Safety and Quality Improvement Act of 2005. As a result of this Act and the Patient Safety Rule, it is expected that some hospitals’ reporting of adverse events could shift in several ways:

• Hospitals that previously did not systematically collect reports of adverse events might start to do so.

• Hospitals might change their data collection process to accommodate use of the common formats.

• Hospitals might use both the common formats as well as their usual methods of reporting.

• Hospitals might report their adverse events to one or more PSOs.

Other concerns that need elucidation include whether mandatory reporting of events have affected decisions to report adverse events to Patient Safety Organizations using common formats and whether and how definitions of adverse events have changed.

History of the Adverse Event Reporting Survey:

The Adverse Event Reporting survey was first developed in 2002 by AHRQ and it's contractor, Westat. Initially, it consisted of two versions of the instrument, one to be answered by risk managers and the other to be answered by six different department managers. The instrument was cognitively tested as part of the questionnaire development process for question and response option wording, and then pilot tested in hospitals to further understand administration procedures. The instrument was reviewed by the American Hospital Association and revised according to their suggestions. While both the risk manager and departmental manager surveys were tested, the risk manager survey provided a relatively complete picture of reporting systems., and could also provide information on what might not be reported to departmental managers. The implementation of the first round of the Adverse Event Reporting survey was conducted by another of AHRQ's contractors, the Rand Corporation, after making minor modifications. In 2005-2006, Rand administered the survey to a nationally representative sample of 2,050 non-federal US hospitals. A second round of the survey administered in 2008 used a random sample of 1,200, selected from the 1,652 hospital Risk Managers who completed the survey in the baseline (1^st round) of the survey.

Plans for future administration of the survey:

Under separate OMB Clearance, AHRQ will plan for a third round of the revised Adverse Event Reporting Survey to be administered in 2011. Information from this survey will allow comparisons over time in the prevalence of reporting, as well as serve as a baseline for evaluating changes as a result of the Patient Safety Act and Rule.

1. Respondent universe and sampling methods

Both data collection activities specified in this information collection request (semi-structured interviews and the cognitive interviews) will use a purposive sample of respondents, and therefore will not be a nationally representative sample. National estimates will not be produced from this redesign effort. However, the lessons learned from this research will inform the redesign of the questionnaire and survey methods. The semi-structured interviews will consist of discussions with seven hospital representatives from seven different hospitals. These hospitals include those not using common formats to report adverse events, those reporting adverse events using common formats to one PSO, and those reporting adverse events using common formats to more than one PSO. The semi-structured interviews will be conducted with the risk manager or other individual familiar with the process of adverse event reporting in each facility. Semi-structured interviews will also be conducted with one representative from each of the two participating PSOs for contextual information on the process of reporting.

The sample of hospitals will include a range of hospitals including rural or urban location; teaching or community hospital; large or small number of beds; or publically or privately held. Individuals participating in the cognitive interviews will be from different hospitals than those participating in the semi-structured interviews. Thirty hospitals will be recruited using a variety of sources, including the contractor’s database of personal contacts in hospitals and recommendations of hospitals from PSOs.

2. Information Collection Procedures

The steps of the survey revision process will include semi-structured interviews, instrument development and revision, cognitive interviews of the revised instrument, and instrument finalization. This process does not require elucidating statistical sampling methodology as it relies more on insights from key informants rather than generalizing to a universe of respondents.

Step 1: Semi-structured interviews. The questionnaire development and testing process will incorporate semi-structured interviews with representatives of hospitals and PSOs to understand the current adverse event reporting environment and how adverse event reporting might have changed for hospitals that previously reported adverse events prior to the implementation of the Patient Safety Act. It is expected that representatives from organizations such as patient safety organizations and hospitals would be able to provide background and contextual information on any changes in reporting practices as a result of the opportunity to report adverse events to a PSO. The topics of these interviews would reflect the different perspectives of these organizations, and focus on the informant’s observed experiences and industry knowledge of changes in adverse event reporting. These interviews would be open-ended and semi-structured, but aimed at describing the processes and outcomes of changes in adverse event reporting. Knowledgeable individuals will be solicited for this part of the instrument redesign by relying on recommendations from stakeholders. These interviews would focus on the following topics:

• Current adverse event reporting activities

• Whether reporting is mandatory

• Use of PSO services

• Use of common formats to report adverse events

• Process of reporting to PSOs, particularly if the hospital contracts with more than one PSO.

• Changes in reporting processes as a result of new relationships with PSOs regarding process of reporting

• Changes in the definitions of adverse events being reported

• Uses of reported information

One person from each of the two participating PSOs will be selected for the semi-structured interviews to provide more context and to understand the changes hospitals must undergo to understand their roles in the reporting process, if any.

The first two rounds of the Adverse Event Reporting Survey targeted the Risk Manager as the respondent. Given possibly large changes in the reporting structure of hospitals with the use of the common formats, questions about whether the Risk Manager is still the most appropriate respondent will be asked.

Step 2. Instrument revision. The current adverse event reporting survey will be revised to reflect the current environment of adverse event reporting. It is expected that one instrument that will accommodate all three reporting scenarios (neither using common formats nor reporting to PSOs; using common formats and reporting to one PSO, and using common formats and reporting to more than one PSO). The instrument will preserve benchmarking information so that trends over time, starting from the first round of the survey, can be reported for comparative purposes.

Step 3. Cognitive testing interviews and revision. Cognitive testing of questionnaires is an important part of the questionnaire development process. Cognitive testing allows a questionnaire developer to understand how respondents answer specific questions, with the goal of determining whether questions are fully understood by the respondent. Features of the question, such as wording, skip patterns, or question order can be explored in the cognitive testing process and allows the questionnaire developer to revise the question based on the testing. A large body of research on cognitive testing supports their wide use in federal and other survey development efforts (see Willis, 1999a).

Two different techniques are traditionally used in cognitive testing: asking the respondent to think aloud as they answer the questions, and using verbal probes. The think aloud technique will be used to test questions that describe a specific activity (see Willis, 1999b) such as the adverse event reporting process. In these cases, respondents will be asked to think aloud about the question and the answer as they respond to the question. To test other questions, verbal probing can be used to get contextual or background information about a question. Verbal probes can be used either during the administration of the question for specific information about a particular question, or after the conclusion of the interview as a kind of debriefing. Both cognitive interviewing techniques will be used depending on the type of question.

The testing will also address whether the Risk Manager is still the most appropriate respondent, or if there is another individual who might be better suited to complete the survey.

The questionnaire will be revised according to the findings of the semi-structured and cognitive interviews, to produce a final version of the instrument. Cognitive interviews will be conducted with one risk manager or other representative responsible for adverse event reporting in 30 participating hospitals. Respondents will represent the three possible reporting scenarios described above: not using common formats to report adverse events, reporting adverse events using common formats to one PSO, and reporting adverse events using common formats to more than one PSO. A recruitment strategy will be developed for respondents which will provide the testing process with perspectives from large, small, urban, rural, teaching, community, publicly and privately owned hospitals.

3. Methods to Maximize Response Rates

The recruitment methods for the formative, semi-structured interviews and cognitive interviews will not require the usual survey methods (repeated mailings, for example) for increasing response rates. Project staff will contact respondents to encourage them to participate.

4. Tests of Procedures

This redesign of the Adverse Event Reporting Survey will use established methods to collect the data necessary to achieve the projects' goals.

5. Statistical Consultants

For this re-design project, James Battles, PhD, Agency for Healthcare Research and Quality provided guidance and expertise in the objectives of the redesigned survey. Representatives of the American Hospital Association extensively reviewed and commented on the first version of the Adverse Event Reporting Survey.

8