Background In The Netherlands, health professionals have created a doctor-driven standardised system to report and analyse adverse outcomes (AO). The aim is to improve healthcare by learning from past experiences. The key elements of this system are (1) an unequivocal definition of an adverse outcome, (2) appropriate contextual information and (3) a three-dimensional hierarchical classification system.
Objectives First, to assess whether routine doctor-driven AO reporting is feasible. Second, to investigate how doctors can learn from AO reporting and analysis to improve the quality of care.
Methods Feasibility was assessed by how well doctors reported AO in the surgical department of a Dutch university hospital over a period of 9 years. AO incidence was analysed per patient subgroup and over time, in a time-trend analysis of three equal 3-year periods. AO were analysed case by case and statistically, to learn lessons from past events.
Results In 19 907 surgical admissions, 9189 AOs were reported: one or more AO in 18.2% of admissions. On average, 55 lessons were learnt each year (in 4.3% of AO). More AO were reported in P3 than P1 (OR 1.39 (1.23–1.57)). Although minor AO increased, fatal AO decreased over time (OR 0.59 (0.45–0.77)).
Conclusions Doctor-driven AO reporting is shown to be feasible. Lessons can be learnt from case-by-case analyses of individual AO, as well as by statistical analysis of AO groups and subgroups (illustrated by time-trend analysis), thus contributing to the improvement of the quality of care. Moreover, by standardising AO reporting, data can be compared across departments or hospitals, to generate (confidential) mirror information for professionals cooperating in a peer-review setting.
Statistics from Altmetric.com
- quality assurance, healthcare
- lessons learnt
- surgical context
- The Netherlands
Most studies on medical adverse outcomes (AO) rely on retrieval and reconstruction of events from patients' records by trained reviewers.1–6 Such approaches have limited effectiveness for quality assessment and improvement. First, retrospective retrieval suffers from the fact that valuable contextual information may be lost when data collection occurs long after the actual medical treatment. Second, older studies often focus on negligence or the type of injury inflicted, and not on learning from past events.2 ,3 ,5 ,7
What is needed, and cannot be delivered by retrospective studies, is a system that encourages healthcare providers to report AOs routinely and then gives immediate and meaningful feedback by which doctors can learn from past events. With these aims in mind, the Dutch Order of Medical Specialists set nationwide standards to create a doctor-driven system of AO reporting that is integrated into routine clinical care. Its key elements are (1) an unambiguous definition of what constitutes an AO, (2) a minimum set of contextual data needed for analysis, and (3) a classification system that organises AOs into clinically and causally relevant groups and subgroups.
The present study elucidates the principles and methods of this system, assesses its advantages and limitations and illustrates its clinical use in the surgical department of a Dutch university hospital.
Patients and methods
A standardised AO reporting system has been in use at the Department of Surgery of the Leiden University Medical Centre since 1997. It is used on all patients who were admitted to, or under the responsibility of, the Department of Surgery, and on patients admitted to non-surgical departments who were operated by a surgeon (or an intern or a resident). Each weekend, a software routine of the Leiden University Medical Centre hospital information system retrieves patients meeting the above surgical criteria who had been discharged in the week that ended 2 weeks before that weekend. The surgical procedures that these patients underwent cover the whole of general surgery, ranging from endocrine, gastrointestinal, oncological, transplant, trauma, to vascular surgery.
Definition of adverse outcome
In The Netherlands, an “adverse outcome” is defined as:
“… an unintended and unwanted event or state occurring during or following medical care, that is so harmful to a patient's health that (adjustment of) treatment is required or that permanent damage results. The adverse outcome may be noted during treatment or in a predefined period after discharge or transfer to another department. The intended results of treatment, the likelihood of the adverse outcome occurring and the presence or absence of a medical error causing it, are irrelevant in identifying an adverse outcome.”
The Dutch definition of an AO is more sensitive than the definitions used in retrospective studies in that those definitions excluded the consequences of the disease itself and/or co-morbidity.1–3 Using a definition with such exclusion rules in routine reporting would require a doctor to completely understand the cause of an AO and its preventability and/or blame. The Dutch definition obviates such judgement, with the aim of reducing subjectivity and inter-observer variability and thereby facilitating comparisons over time and/or across different settings.
Management of (contextual) information
A special software was designed to support the management of hospital information system and AO data. All the patients identified by the hospital information system search were automatically imported each week into an AO database (Access 2003) with a minimum dataset of contextual information, including name, gender, date of birth, operative procedures performed and the American Society of Anesthesiologists (ASA) classification. Operative procedures were automatically classified with respect to technical complexity, according to the standards of the Association of Surgeons of The Netherlands (on a scale ranging from 1 for simple to 5–7 for complex or difficult surgical procedures).
The occurrence or non-occurrence of all AO was recorded on paper for all patients. Selected AOs were checked and discussed at weekly meetings attended by surgeons, residents and interns and chaired by a staff surgeon. The selection of AOs was based on an algorithm that included patients (1) with three or more AOs, (2) with one or more AOs leading to reoperation, permanent damage and/or death (ie, grade 2 or higher; see below) or (3) with an AO report that seemed likely to contain a lesson for the future. The aim of the above selection was to ensure interesting meetings and thus maximise attendance. Each patient with an AO was presented by the intern/resident who was on the ward when the patient was discharged. The focus of the ensuing discussion was to interpret the underlying causes and thus learn how other such outcomes might be prevented. After the meeting, data forms of all patients, without and with AOs, discussed and non-discussed, are entered into the AO database using the software described above. Patients with one or more AOs are the denominator of the proportion of patients with an AO, the whole patient set being the numerator.
AOs were classified according to a special classification system that was designed to match existing (inter)national classification systems as closely as possible. This “master classification” facilitated analysis by allowing AO to be automatically aggregated (by the software) into specific groups and subgroups, The master classification system defines the key characteristics of each AO in terms of the following three dimensions (see Online Appendix A):
Type of AO that is its pathology (eg, infection, bleeding)
Location, specified by
a. region (eg, thorax, abdomen) and/or by
b. organ or organ system (eg, lung, liver) and/or by
Contextual information and potential determinants (eg, medication).
The first dimension—type—is based on the International Classification of Diseases, 10th revision, but with adaptations to accommodate AOs (as opposed to spontaneous disease). The second dimension—location—is based on the Dutch WCC Standard Anatomical Location (http://www.rivm.nl/who-fic/Annuals/Paris26.pdf), developed by WHO. Of the various elements that fall under contextual information dimension, the coding of medication was derived from the Anatomical Therapeutic Chemical classification (http://www.whocc.no/atc_ddd_publications/guidelines/). Each dimension comprises two or more degrees of detail, to allow general and detailed analyses. AOs were graded according to four levels of severity: (1) temporary health disadvantage, recovery without (re)operation; (2) recovery after (re)operation; (3) (probably) permanent damage or function loss; and (4) death.
AOs were analysed in two ways: by discussion of individual patients at weekly AO meetings and by regular statistical analysis of patient groups and subgroups.
Individual patients were discussed at by means of four structured questions shown below (table 1). The answers given provided (interpretational) information on determinants and preventability. All answers generated since 2003 have been stored in a second database. Lessons learnt can lead to quality improvement on for individual patients and for similar patients in the future.
The statistical analysis of the groups and subgroups was performed at regular intervals, at least once a year or whenever deemed necessary—for example, a clinical query or a research question. Logistic regression analysis was used to study the relation between patient mix and operation mix on the one hand, and the incidence of (admissions with) AOs on the other. Patient-mix characteristics were age at admission, gender, and anaesthetic risk profile (high risk being defined as ASA 3 or higher at first operation).6 ,8 The operation mix consisted of various degrees of surgical technical complexity or difficulty.
For the analysis of time trends, the total study period of 9 years was subdivided into three equal periods (P1: 1997–1999, P2: 2000–2002 and P3: 2003–2005), which allowed for a more robust analysis than analysis per year. Logistic regression analysis was carried out for all 18 types of AOs and for each of the four grades of severity. The number of AOs per subset divided by the number of all AOs was taken to be an indicator of a subset's contribution to the overall trend. If the incidence of a specific type of AO changed/increased significantly over time to such an extent that it contributed more than 5% of all AOs, further analysis was performed to identify which subgroups within that category, defined on the second or third dimension, contributed most to the observed time trend.
In all analyses, significance was determined by p values smaller than 0.01, given the large number of patient admissions.
Over the 9-year study period, 19 907 patient admissions met the inclusion criteria for surgical AO reporting. For all these admissions, the absence or occurrence of AOs was documented and verified. Mean patient age at admission was 46.7 (21.9) years, 47.5% of patients were women, 18.5% were ASA 3 or higher and 29.9% underwent “complex” surgery (see table 2). A total of 9189 AOs were reported, with one or more AOs occurring in 18.2% of admissions (table 2).
Just under half of all AOs (44.5%) were discussed at weekly meetings. Most AOs were deemed to be related to operative treatment (42.5%), followed by non-surgical treatment (29.4%), primary disease (14.5%) and co-morbidity (13.5%). From 2003 to 2005 (P3), 20.9% of all AOs were considered (potentially) avoidable and 4.3% led to specific “lessons learnt”; that is, an average of 55 lessons per year.
Older patients had more AOs, as did high-risk patients and patients undergoing complex surgery (table 3). The overall incidence of reported AOs increased over time (table 3), AO incidence being 39% higher in P3 than in P1, mainly because of an increase in minor AOs (severity category 1, see table 4). The incidence of AOs of severity category 4 (death of the patient) decreased by 40% between 1997 and 2005.
Investigation of AO types showed that 7 out of 18 main-type categories occurred more frequently in P2 and/or P3 than in P1. The two categories that occurred most frequently (indicated in bold in table 5) were “inflammation/infection” and “functional disorders” (29% and 23%, respectively; other categories only 7%).
Investigation of AO location showed that AOs of the abdomen, leg (table 6) and circulatory tract (table 7) contributed most to the observed increase in “inflammation/infection”. Investigation of the third dimension—context—revealed that wound infections increased significantly (odds ratio (OR) 1.54 (1.20–1.98) for P3 vs P1) and accounted for 27% of the overall infection increase. Circulatory tract infections, in particular those related with the use of vascular catheters, likewise increased (OR 2.97 (1.51–5.82) in P3 compared with P1) but contributed only 4.4%.
The increase in “functional disorders” was mainly explained by a significant increase in functional disorders of the kidney (OR 2.32 (1.52–3.52) for P3 vs P1, and contributing 12.8% of functional disorders), and by medication-related functional disorders (OR 2.32 (1.06–5.06) for P3 vs P1).
The present study demonstrates that doctor-driven reporting of AO is feasible alongside routine clinical care over a long period. Even after adjusting for shifts in patient mix and operation mix, it was seen that the number of non-serious AO reported increased rather than decreased, whereas the number of deaths (severity 4 AO) decreased. This suggests that the reporting of AO has become increasingly sensitive over time and that surgical care has become safer with respect to mortality. AO reporting was used to support professionals in assessing and improving their own quality of care in two ways. First, through the weekly AO meetings, surgeons discuss and analyse AOs that their patients suffered case by case. The emphasis lies on ‘What can we learn from this?’ instead of on ‘What have they done wrong?’. The lessons learnt in this way averaged 55 per year in the third period, and applied to many areas ranging from thrombosis prophylaxis in subgroups of patients to technicalities of fracture management at the operating room, and from the appropriate use of antibiotics in trauma patients to timing and preoperative workup of liver transplant patients. Second, at a group level, the aggregation of AOs into master classification groups makes it possible to carry out a statistical analysis of AOs and possible determinants, independent of the weekly case-by-case analyses. As an example, the time-trend analysis presented here demonstrates a significant increase in functional disorders of the kidney, which is most likely related to new immune-modulating drugs used in transplantation and an increase in reported infections. A detailed analysis of the latter has been published in a separate paper.9 Infections identified by AO reporting differ from those identified by monitoring of the Centers for Disease Control and Prevention, as AO reporting includes all suspected or proven infections that do not resolve without treatment, that is, that require(d) treatment or have been treated. Apart from the fact that infection monitoring by the Centers for Disease Control and Prevention is not ubiquitous in all Dutch hospitals, it does not routinely provide detailed information on context or potential causes.
Outside the context of the registering department or hospital, doctor-driven AO reporting can also be used for “mutual monitoring”. When professionals from different departments or hospitals cooperate through a (confidential) national AO database, this can provide mirror information to allow doctors to compare their own AO data with anonymous AO data that have been collected by professionals working in similar hospitals or departments. For such a purpose, the system described here has been adopted as the national standard for AO reporting by the Association of Surgeons of The Netherlands. Obviously, additional important issues will need to be addressed, such as adjusting for under-reporting and for more detailed differences in patient and operation mix, to further improve the validity of mirror comparisons.
The Dutch Healthcare Inspectorate analyses the quality of care of all Dutch hospitals annually through national “external” benchmarking of a set of 20 quality performance indicators. One of these indicators is whether medical professionals and hospital departments practice AO reporting. The Inspectorate has, in line with recommendation 6.1 of the Institute of Medicine report “To err is human”i on peer-review protections, stated that AO data are and will at all times remain the sole property of the medical professionals and will never directly be used for benchmarking. However, if professionals are found to have out-of-range performance indicators, they themselves can use the detailed information provided by the AO reporting system to bring their performance up to standard.
Other research has been published on how to make healthcare safer.10–12 Chang et al13 proposed a standardised terminology and classification for adverse events. The approach presented here differs from Chang et al's in that it is not limited to outcomes that are related to errors in the medical (management) process. By eliminating causality, such as errors, as a criterion for reporting, the “Dutch approach” is more sensitive and has less observer-dependent reporting. In addition, it provides information about all AO and is thus a more relevant source of information because it does not exclude AOs that are not caused by errors. The fact that Dutch AO reporting is integrated into routine clinical practice has advantages over other initiatives that are not rooted in routine patient management, such as the National Confidential Enquiry into Patient Outcome and Death.14 ,15 Not the least of these advantages are the regular plenary discussions among clinicians and the lessons learnt directly each week.
Doctor-driven reporting inevitably has drawbacks as well, the most obvious being underreporting. However, we have shown in an earlier study that, with the overriding focus on learning, underreporting can be kept to a minimum.
The internal monitoring described here, and the mutual monitoring, which is presently part of an initiative in The Netherlands, demonstrate that doctor-driven AO reporting can be a powerful instrument with the potential to prevent AO in patients in the future.
The definition and the classification system presented here, after being pioneered at the Department of Surgery at the Academic Hospital Leiden in the 1980s by the first author (JK), have benefited from more than two decades of input from many individuals and organisations, including the Association of Surgeons of The Netherlands, the Committee on Adverse Outcome Reporting of the Order of Medical Specialists and Prismant (http://www.Prismant.nl). Their important contributions are gratefully acknowledged. Further information can be obtained from the Order of Medical Specialists, Utrecht, The Netherlands (http://orde.artsennet.nl/), or through the authors. Finally, the authors gratefully acknowledge the editorial help of Mrs Petry Kievit-Tyson.
Funding Over the years funding for the nationwide development of the surgical adverse outcome registration was obtained from the Health Research and Development Council ZONMW (by grant number 16450001), from the Association of Surgeons of The Netherlands, and from the Order of Medical Specialists.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
↵i See Recommendation 6.1. Congress should pass legislation to extend peer-review protections to data related to patient safety and quality improvement that are collected and analysed by healthcare organisations for internal use or shared with others solely for purposes of improving safety and quality.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.