BENEFITS OF INCIDENT REPORTING

Political initiatives have encouraged the development of incident reporting systems both in the UK and the US.^1,2 These have been partly motivated by the apparent success of incident reporting in the field of aviation following patterns established by NASA’s Aviation Safety Reporting System and the UK Confidential Human Factors Incident Reporting scheme (CHIRP). The focus moves away from the analysis of low frequency, high consequence events to more frequent near miss events. This offers numerous benefits. Incident information provided by pilots, ground crew, and air traffic managers helps regulators to find out why accidents don’t occur. Reports describe how operators cooperate to detect and mitigate adverse events. The higher frequency of incidents also permits quantitative analysis in a way that is not possible with lower frequency accidents. Information about previous incidents is, typically, published in newsletters and increasingly on web sites. This reminds staff about potential hazards and helps to keep them “in the loop”. The data (and lessons) from incident reporting schemes can be shared. Incident reporting systems provide the raw data for comparisons both within and between industries. If common causes of incidents can be observed, it is argued that common solutions can then be found. Incident reporting schemes are cheaper than the costs of an accident. A further argument in favour of incident reporting schemes is that organizations may be required to exploit them by regulatory agencies. There is also a widespread but erroneous perception that these systems are “low cost” and simple to operate.³ Given this list of benefits in the aviation domain, it is hardly surprising that incident reporting should be proposed as a key technique to improve patient safety.

PROBLEMS OF PARTICIPATION BIAS

A number of problems complicate the implementation and operation of reporting systems. It can be difficult to gather incident reports from medical staff. Those reports that are obtained may not be well distributed across all grades of staff. For example, a Scottish adult intensive care unit found that nursing staff had contributed 90% of all reports submitted over the last decade, with nurses submitting 621 reports and medical staff reporting only 77 incidents.⁴ This bias is often assumed to indicate reluctance by senior staff to participate in reporting schemes. As a result, schemes will pay undue attention to the execution of medical procedures by nursing staff rather than the planning, coordination and administration of treatment by senior personnel. However, things are more complex than this simplistic analysis would suggest. It is difficult to interpret the distribution of reports from different clinical disciplines and grades. It is important to consider the total number of staff who might contribute to such a system. Usually the team consisted of three medical staff, one consultant, and up to eight nurses per shift. The larger number of reports contributed by nursing staff can also be explained in terms of the involvement in, or exposure to, the types of workplace incidents that were solicited under this particular scheme. Nursing staff had the most direct contact with the patients who remain the focus of the reporting system. Hence, it can be argued that they have a proportionately greater opportunity to witness adverse events.

PROBLEMS OF ELICITATION AND FORM COMPLETION

The problems of eliciting incident reports can be illustrated by a recent comparison between voluntary incident reporting, the computer based analysis of patient records, and exhaustive manual comparisons of patient data looking for adverse drug events.⁵ When admissions to nine medical and surgical units in an 8 month period were studied, the computer monitoring strategy identified 2620 incidents of which 275 were confirmed to be adverse drug events. The manual review found 398 adverse drug events while voluntary reporting only detected 23. It is too simplistic to argue that this disparity stems from a reluctance to make voluntary reports. It can be difficult for staff to detect incidents when they occur. Adverse consequences may only emerge over a matter of days, weeks or months. It can also be difficult for busy clinical staff to find the time to complete a report for minor incidents, for mishaps that they are unsure about or where issues of anonymity are important. There are also more practical problems; in particular, printed reporting forms are often difficult to obtain. Staff must be motivated to find one, fill it in, and then post it to the appropriate safety managers. Many national schemes have responded to these practical problems by introducing web based systems. However, most of these applications are deeply flawed in terms both of their engineering and, in particular, their usability. Typically, they have large logos announcing that the user is making a submission through the voluntary reporting system. For many medical staff this can be very off putting if they have to submit a form using a shared PC in a busy ward. Further problems stem from the design of these new computer based reporting forms. For example, one system forced the user to enter the date of an incident using pull down menus entitled date, month and year. They were also forced to specify a single time at which the incident occurred. This created enormous problems when usability tests were conducted with healthcare staff. They found it difficult to enter information about incidents such as those where a medication had been incorrectly administered several times. Should they enter the first date when this occurred or the date on which it was identified? One group of clinicians even entered six separate forms for each instance, thereby creating problems for the safety managers who must identify that they all involved the same patient. Many of these problems affected the design of previous generations of paper based reporting forms. However, computer based systems are less forgiving in terms of the ad hoc annotations and inclusions that can be submitted about adverse events and near misses.

PROBLEMS OF ANALYSIS AND CLASSIFICATION

Assuming that the new generation of reporting systems can overcome the barriers to submission, they must face a number of additional problems associated with the analysis of adverse clinical events. The US Aviation Safety Reporting System relies upon dedicated teams of coders who will analyse each incident submitted to the system. This is appropriate because coders can be trained and monitored to ensure consistency in their work. Unfortunately, the analysis of adverse events illustrates further differences between the aviation and healthcare domains. In particular, it is unlikely that any national system could afford to employ dedicated teams of coders for all medical events. The Aviation Safety Reporting System spends about $3 million annually to analyze approximately 30 000 reports. This equates to about $100 per case. At this rate, it would cost around £50 million to conduct a similar level of analysis for the 850 000 adverse events that are estimated to occur each year within the NHS.⁶ It has been estimated that the cost of clinical negligence to health authorities and NHS trusts was approximately £200 million in 1995–1996. The cost of claims is expected to rise by 20% per annum over the next 5 years. The NHS summarized accounts for this period include provision totalling £80 million with contingent liabilities of £1.6 billion. The costs of a centralized analysis service have persuaded many regulators to encourage local investigation by safety managers. This is a sensible approach given that these individuals often have a clear understanding of the context in which an adverse event occurred. However, in the UK they lack standardized training. In a series of empirical studies it has been shown that there is no agreement over the causal analysis of a sample of adverse events by safety managers within the NHS.⁷ This has led to the development of analytical tools and classification schemes, similar to flowcharts that help safety managers to identify appropriate causes from the incident information. Unfortunately, previous experience has shown numerous limitations with these techniques. There are theoretical problems—for instance, the more exhaustive the taxonomy then the more likely it is that analysis will find a classification that matches their incident. It is also likely, however, that the use of an extended classification will reduce the consistency of any analysis of similar events by different investigators. The FDA describes further problems. A violent patient in a wheelchair was suffocated through the use of a vest restraint that was too small. The risk manager (JC) continued:

“She finds the list of event terms, which was detached from the rest of the coding manual ... She muses: ‘Mr. Dunbar had OBS which isn’t listed in these codes; he had an amputation which is listed; he had diabetes which isn’t listed; and he had hypertension which is listed’. JC promptly enters 1702 (amputation) and 1908 (hypertension) in the patient codes. She then finds the list for device-related terms ... She reviews the terms, decides there was nothing wrong with the wheelchair or the vest restraint, and leaves the device code area blank.”⁸

The resulting classification of 1702 (amputation) and 1908 (hypertension) provided few insights into the nature of the incident. It is, however, characteristic of the analysis obtained from overstretched staff faced with complex incidents and inadequate training.

LACK OF COMPUTATIONAL EXPERTISE

Several national governments have commissioned national patient safety agencies to encourage the reporting of adverse events systems. A key element in this national infrastructure is the provision of software tools to store the many thousands of incidents analyzed by regional safety managers. Unfortunately, there has been a lack of investment in appropriate technologies to support this task. This shortcoming can be explained in terms of a reliance on human factors experts rather than software engineers in the design and development of these systems. The consequences of this lack of engagement are far reaching. Poor user interfaces to many electronic reporting forms have already been mentioned. The lack of computational expertise also affects the design of storage and retrieval systems for incident records. Most of the proposed national schemes are based around relational database technology. In essence, this stores incident data according to the classification schemes that have been developed by human factors experts to classify the causes of human error in terms, for instance, of slips, lapses and mistakes. However, problems arise when the taxonomies change. This is likely when the human factors view of key concepts such as workload have developed radically over the last decade. The net effect is that, in 10 years time, we may have to go back into our electronic databases and manually reclassify many hundreds of thousands of reports to reflect a revised taxonomy.

PRECISION AND RECALL

There are further technical issues in the development of reporting systems. Precision and recall are concepts that are used to assess the performance of all information retrieval systems. The precision of a query is measured by the proportion of all documents that were returned which the user considered to be relevant to their request to the total number of documents that were returned. In contrast, the recall of a query is given by the proportion of all relevant documents that were returned to the total number of relevant documents in the collection. It therefore follows that some systems can obtain high recall values but relatively low precision. In this scenario, large numbers of relevant documents will be retrieved together with large numbers of irrelevant documents. This creates problems because the user must then filter these irrelevant hits from the documents that were returned by their initial request. Conversely, other systems provide high precision but poor recall. In this situation, only relevant documents will be returned but many other potential targets will not be retrieved for the user. In most other areas of software engineering, the trade off between precision and recall can be characterized as either performance or usability issues. In incident reporting schemes these characteristics have considerable safety implications. For instance, low recall systems result in analysts failing to identify potentially similar incidents. This entirely defeats the purpose of compiling national and international collections. Conversely, low precision approaches leave the analyst with an increasing manual burden as they are forced continually to navigate “another 10 hits” to slowly identify relevant reports from those that have no relation to their information needs. Again this can result in users failing to accurately identify previous records of similar incidents. There has been relatively little investment by patient safety agencies in these issues. In consequence, it can be argued that we are relatively unprepared for the problems that will lie ahead when increasing amounts of data are obtained about adverse events and near miss incidents.

UNREALISTIC EXPECTATIONS AND POLITICAL PRESSURE

Charles Billings, the former Chief Scientist at NASA Ames, has identified some of the limitations that might affect healthcare schemes based on his experience with the aviation safety reporting system:

“... there are enough reports of mishaps with potassium chloride, lidocaine, vincristine and other drugs and devices to have made it very clear that a problem with these exists. The information that these events occur is already present. We may well ask what it is that keeps us from making progress on safety, given that we already know about the existence of these problems. What is added by more formal, elaborate (and expensive) incident reporting?”⁹

This is a crucial insight. National governments have already established high targets and aspirations for their healthcare systems. For instance, the NHS has been asked to meet the following objectives:

“... by 2001, reduce to zero the number of patients dying or being paralyzed by maladministered spinal injections (at least 13 such cases have occurred in the last 15 years); by 2005, reduce by 25% the number of instances of negligent harm in the field of obstetrics and gynaecology which result in litigation (currently these account for over 50% of the annual NHS litigation bill); by 2005, reduce by 40% the number of serious errors in the use of prescribed drugs (currently these account for 20% of all clinical negligence litigation); by 2005, reduce to zero the number of suicides by mental health inpatients as a result of hanging from non-collapsible bed or shower curtain rails on wards (currently hanging from these structures is the commonest method of suicide on mental health inpatient wards).”¹⁰

It is unlikely that incident reporting alone will yield the insights necessary to achieve these objectives. Incident reporting systems often only serve to confirm existing suspicions about the causes of medical errors. They do not suggest immediate solutions. Fortunately, these schemes provide one component of broader initiatives to address healthcare incidents. National patient safety organizations are commissioning research in many different areas. These will, arguably, have a greater impact than the reporting systems. For example, smart infusion devices are being designed to recognize the drugs that they administer using indirect sensing techniques. Other initiatives are focusing on device labelling and communication protocols between teams of healthcare workers. Few of these initiatives stem uniquely from the insights of reporting systems.

CONCLUSION

Incident reporting systems do yield good insights into local problems and in ideal situations can be used to identify regional and national patterns of failure. However, my intention has been to encourage a critical reappraisal of the claims that are being made for these systems. A secondary aim has been to encourage national patient safety agencies to look beyond the teams of human factors specialists who are developing taxonomies of medical error. In particular, it is essential to recruit user interface designers who can help clinicians and safety managers to enter information about complex adverse events—for example, through direct user testing of on line forms. Similarly, it is important to recruit specialist expertise in the storage and retrieval of large data sets. There are particular attributes of incident reporting systems that make it dangerous to rely on conventional relational databases. Alternative techniques, including free text retrieval, offer greater flexibility and support the use of human error taxonomies that will change over time. It should be no surprise that these are key areas for recent investment within many of the major aviation reporting systems.³ It would be better for patient safety agencies to invest now rather than relearn the lessons of existing reporting systems in other domains.