Article Text

Download PDFPDF

The use and impact of national confidential enquiries in high-income countries
  1. Aniela Angelow1,
  2. Nick Black2
  1. 1Institut for Community Medicine, Universitat Griefswald, Griefswald, Germany
  2. 2London School of Hygiene & Tropical Medicine, London, UK
  1. Correspondence to Professor Nick Black, Department of Health Services Research and Policy, London School of Hygiene & Tropical Medicine, 15–17 Tavistock Place, London WC1H 9SH, UK; nick.black{at}lshtm.ac.uk

Abstract

Objective To describe the use and characteristics of national confidential enquiries (NCEs) into adverse outcomes of healthcare in high-income countries and to review the evidence of their impact.

Method Systematic search of bibliometric databases plus review of cited references and search of websites. Eleven characteristics of NCEs were extracted. Studies evaluating the impact of three NCEs were searched for. Data were extracted and tabulated, and a narrative review conducted.

Results Establishment of NCEs has been limited with only 27 examples identified in over 50 years and only nine currently functioning. They have been particularly popular in the nations of the UK (17 of the 27) and in services around childbirth (15/27). NCEs mostly include all cases (19/23) and include adverse outcomes both during and after the initial hospital episode (17/23). The annual volume of cases varies from four to over 6000. With one exception, NCEs make no attempt to use ‘controls.’ Research evidence of the impact of the recommendations from three of the largest and longest running NCEs is poor, with no time-series analyses or experimental studies, and is restricted to considering their impact on the structure and process rather than the outcome of care.

Conclusions The lack of scientific evidence on the impact of NCEs on improving safety, combined with uncertainty as to the validity of their recommendations and their high cost, suggests the need for rigorous evaluation and a reconsideration of their contribution. One option is to nest NCEs within prospective national clinical audits.

  • Clinical practice guidelines
  • guideline
  • mortality
  • adverse event
View Full Text

Statistics from Altmetric.com

Introduction

While guidance on the effectiveness of healthcare relies on evidence from research, guidance on how to improve the safety of care has tended to rely heavily on evidence from reviews of patients who suffered an adverse outcome. The longest established approach has been the confidential enquiry, first used locally in the 19th century in the UK but extended to a national scale in the 1920s, for maternal mortality. In 1952, to enhance participation, the first well-recognised national confidential enquiry (NCE), the Confidential Enquiry into Maternal Deaths, was established in England and Wales with the aim of identifying common causes of maternal deaths and making recommendations to improve the safety of clinical practice.1 Its principal feature was the systematic review by independent experts of individual cases in which the patient died. The identity of the cases and of the experts remained confidential, with only aggregated findings based on many cases being made public.

Over the years, although the key features of the method have remained unchanged, some variations in the approach adopted by NCEs have been introduced: one-off enquiries versus ongoing or repeated; inclusion of all cases versus sampling; consideration of adverse cases only versus the additional inclusion of ‘controls’; use of a single expert reviewer versus more than one; use of predefined assessment criteria versus implicit judgements; and maintaining complete confidentiality versus identification of service providers or practitioners.

Despite widespread professional support, the extent of the use of NCEs in high-income countries and the impact of their recommendations on the safety of healthcare is unclear, as is their cost-effectiveness compared with other ways of establishing guidance on safety. Our objectives were to describe the use and methodological characteristics of NCEs into adverse outcomes of healthcare in high-income countries, to review the evidence of their impact on the safety of healthcare by considering the three longest established ones in the UK (Confidential enquiry into Maternal Deaths; Confidential Enquiry into Stillbirths and Deaths in Infancy; National Confidential Enquiry into Patient Outcome and Death), and to make recommendations about the contribution of NCEs to stimulating improvement in safety in high-income countries today.

Methods

Use and methodological characteristics of NCEs

To be included, a confidential enquiry had to be national in scope and focused on adverse outcomes of healthcare. Relevant electronic databases were searched, employing the free text terms ‘confidential enquiry’/‘confidential enquiries’/‘confidential inquiry’/‘confidential inquiries’ in the title, abstract and/or keywords. Where the abstract could not be searched, the document's full text was searched in addition to the title (POPLINE). Because of the diversity of topics covered by NCEs, the search strategy was not refined using further text or MeSH terms. No language and date restrictions were used.

An additional search included official governmental websites of selected countries (UK, USA, Australia, New Zealand, Canada, Israel) identified through key publications, institutional websites (restricted to countries with English as a main language), the Index To Thesis database (restricted to titles) and the internet via Google (restricted to a search of pdf documents).

One author (AA) screened titles and abstracts of retrieved references. The full reference was obtained if insufficient information for classification was possible. Where possible, the most recent official NCE report was obtained. If an official report was not available, information was extracted from journal articles. Each NCE was assessed according to 11 characteristics (box 1) using a standardised form. Descriptive analyses were performed to obtain a summary account.

Box 1 Characteristics of national confidential enquiries

  • Topic

  • Country

  • Date commenced and ended

  • Inclusion of all cases or a sample

  • Inclusion of cases experiencing adverse outcome after discharge from hospital

  • Number of cases

  • Inclusion of controls

  • Use of explicit criteria/checklist

  • Case assessment by a multiprofessional panel

  • Production of recommendations

  • Feedback on personal practice to providers

Impact of three national confidential enquiries in the UK

The three most longstanding NCEs in the UK were selected: the Confidential Enquiry into Maternal Deaths (CEMD), the Confidential Enquiry into Stillbirths and Deaths in Infancy (CESDI)—both incorporated in the Confidential Enquiry into Maternal and Child Health since 2003—and the National Confidential Enquiry into Patient Outcome & Death (NCEPOD), until 2002 limited to perioperative deaths. The search included a wide range of study designs for evaluating their impact: randomised controlled trials, non-randomised studies (time series, cohort, case–control, before and after), cross-sectional surveys and qualitative studies were included. The impact of NCE recommendations was considered in terms of the impact on: clinicians' knowledge and attitudes; healthcare structure (staffing, facilities); processes of care; and health outcomes (mortality and morbidity).

The title and abstract of retrieved references were assessed for eligibility. Where information was incomplete, the full record was obtained and assessed.

A standardised data-extraction sheet was used to ascertain information on the study period, design, participants, sampling method, participation rate, data-collection method, impact/s investigated (structure; process; outcome) and the method of assessing the impact (opinion; organisational reports; quantitative measurement). Findings were tabulated and a narrative review conducted.

Results

Search results

The electronic database searches identified 2225 potentially relevant documents. After excluding 1197 duplicates, the titles and abstracts of the remaining 1028 were screened. This resulted in 775 that required a more thorough review, which resulted in the identification of 23 NCEs2–24 that met the inclusion criteria. An additional three NCEs25–27 were identified from reviewing the references of the selected documents, and one NCE28 was identified by an expert adviser. Of the resulting 27 NCEs, official reports produced by those managing the enquiry were available for 16, but for the other 11 we had to rely on accounts in journal articles.

Number and characteristics of national confidential enquiries

Of the 27 NCEs, 17 were undertaken in the UK, six in other European countries (Netherlands, France, Austria) and four outside Europe (Australia, New Zealand, Bahrain, Japan) (table 1). The commonest topics were associated with childbirth: maternal mortality (9), maternal morbidity (2) and perinatal mortality (4). Other topics included adverse outcomes of surgery and anaesthesia (mortality; complications) (6), and mortality associated with respiratory disease (3), epilepsy (1), treatment of heroin addiction (1) and suicide/homicide by people with mental illness (1).

Table 1

National confidential enquiries (in chronological order by start date)

Following the establishment of the CEMD in England & Wales in 1952, only two more NCEs were set up in the following 20 years, both focused on maternal mortality (Australia 1964; Scotland 1965). Of the nine NCEs running in 2009, one was initiated in the 1950s, one in the 1960s, two in the 1980s, three in the 1990s and two since 2000.

Methodology of national confidential enquiries

Information on the methodology of four of the 27 NCEs was limited by the lack of detail in any publications. Of the 23 for which adequate information was obtained, 19 included all relevant patients, and four were based on a sample. The use of sampling was not associated with any particular topic. While 17 NCEs included adverse outcomes occurring during and after a hospital episode, the other six were restricted to during a hospital stay. The mean number of cases included each year varied from four to 6571. Only three have ever included controls.2–4

In the 19 NCEs for which information was available, 17 used explicit criteria for reviewing cases, and 13 were conducted by a multiprofessional panel. As regards output, most (14 of 21) not only described the cases studied but also provided recommendations about how to improve the structure of services (eg, required facilities, staff) and/or the clinical management of patients. Only two provided personal feedback to participants about the enquiry's judgement of clinical performance.5 6

In all NCEs, confidentiality of patients and practitioners was ensured by anonymisation of data. Although only one NCE explicitly stated that all data were destroyed after completion of the enquiry,3 this is understood to be common to most, if not all, enquiries.

Impact of three NCEs on the quality of care

Of the 775 documents identified in the search, 477 referred to the three NCEs that were the subject of the impact review. Of these, only 12 met our inclusion criteria (table 2). Four related to NCEPOD, six to CEMD and two to CESDI.

Table 2

Studies included in a review of impact of three national confidential enquiries (Confidential Enquiry into Maternal Deaths (CEMD), Confidential Enquiry into Stillbirths and Deaths in Infancy (CESDI), National Confidential Enquiry into Patient Outcome & Death (NCEPOD))

As regards study design, eight were cross-sectional,29–36 three used a before-and-after comparison,37–39 and one was a qualitative study.40 One of the cross-sectional studies34 replicated an earlier one,33 enabling a before-and-after comparison to be made. The studies considered the impact of recommendations from NCEs in terms of: clinicians' awareness; changes to the structure of care; and changes to the process of care. None of the studies looked at the impact on outcomes.

Clinicians' awareness of recommendations

The most basic impact, that of clinicians' awareness of, and attitudes to, NCE recommendations, was the subject of two studies. Foy and colleagues interviewed 201 obstetricians and midwives in Scotland about their awareness of the key recommendations from the CEMD report for 1994–1996.30 While 92% were aware of the report, 52% had read some part of it, and on average, clinicians could recall three of the report's 18 recommendations. Greater benefit was reported by 18 clinicians who had reviewed cases for the CESDI.40 They felt that the experience had had a positive impact on their own practice, though they were frustrated that the method did not include feedback of the enquiry's judgement of specific cases to the particular hospital where the death had occurred.

Impact on structural aspects

Seven studies have investigated the impact of recommendations on structural aspects of healthcare such as staffing levels, availability of facilities and the local introduction of guidelines. Four studies were simple cross-sectional surveys. In 1995, Derrington and colleagues sought the opinion of 100 anaesthetists about the impact of NCEPOD recommendations.29 Of the 72 who responded, three-quarters reported that the recommended guidelines had been introduced in their hospital, and there had been improvements in staffing and equipment. There was greater uncertainty as to the impact of a CEMD recommendation that obstetric anaesthetists should have the support of a trained assistant, as no information was provided as to how many of the 39 (out of 44) already had such support before the recommendation had been published.31 In a much larger study, 2 years later, 40% of 2195 anaesthetists and surgeons reported that NCEPOD recommendations had been influential in improving available facilities.36 This was similar to the results of an earlier study that found variable adherence to recommendations about postoperative recovery facilities and monitoring equipment.32

The limitation of cross-sectional studies was overcome by two before-and-after studies. In 1993, Hibbard surveyed 248 maternity units to check on the availability of CEMD recommended facilities and the adoption of clinical guidelines.33 Three years later, Benbow repeated the study and found some improvement: on-site blood transfusion services had risen from 87 to 96%, on-site intensive care unit from 76 to 80% and availability of clinical guidelines for major haemorrhage from 82 to 93%.34 The other study surveyed 17 hospitals about the availability of NCEPOD recommended facilities in 2001 and again a year later.38 Some improvements were found, including the availability of an emergency theatre (increased from eight to 13 hospitals) and on-call surgeons being free of other commitments (from three to five hospitals).

Impact on processes of care

Two of the cross-sectional studies that considered the impact on structures also considered the impact on processes of care. The survey of 44 obstetric anaesthetists found that adherence to CEMD recommended practice varied considerably, from 100% for the application of cricoid pressure with intubation to 14% for the application of two-handed cricoid pressure.31 Pryce and Coles also reported a variation in clinicians' reports of the impact of different NCEPOD recommendations, ranging from 53% for postponing out of hours surgery to 14% for requesting more post-mortems.36

A third cross-sectional study investigated the impact of recommendations from CESDI in 12 maternity units. Clinicians responsible for risk management reported that only some recommendations had been adopted, partly because of not receiving information on local deaths from the NCE and also because of insufficient time and staff resources.35

The limitation of such cross-sectional studies was overcome by Bolton and colleagues, who carried out a before-and-after study of the impact of the CEMD recommendation about the use of syntocinon in caesarean sections.39 The proportion of obstetric anaesthetists reporting adherence to the recommendation rose from 13 to 85%. Of those reporting a change in practice, 95% attributed it to the CEMD recommendation. In contrast, another before-and-after study conducted in a single hospital which looked at the impact of a NCEPOD recommendation that patients with a fractured femur should undergo preoperative echocardiography, claimed that this harmed patient care by increasing the incidence of operative delays of 3–10 days from 14 to 21% (though not statistically significant given the small study size).37

Discussion

Main findings

The establishment of national confidential enquiries has been quite limited in high-income countries, with only 27 examples identified in over 50 years and only nine currently functioning. They have been particularly popular in the nations of the UK (17 of the 27) and in services around childbirth (15/27).

Although NCEs vary methodologically, most have aimed to include all patients suffering an adverse outcome (18/27), including those that occur after an episode of care (16/23). The annual volume of cases included varies depending on the national population and the incidence of adverse outcomes. With three exceptions, NCEs have been case series, with no comparative data available from ‘controls,’ and, to maintain confidentiality, NCEs have provided only aggregated national data rather than data related to specific providers (hospitals, practitioners).

There is very limited evidence as to the impact that NCEs' recommendations have had on improving the safety of healthcare. Of the 12 studies identified, nine adopted a cross-sectional descriptive design, relying on clinicians' views. They varied in their conclusions as to whether or not the recommendations had had any impact on the structure and process of care, and none attempted to assess the impact on health outcomes. Similarly, the three analytical studies, which used a before-and-after design, suggested that changes in the process of care were sometimes stimulated by NCE recommendations.

Methodological limitations

Given that some NCEs produce only grey publications, we may have failed to identify some. For some of the 27 identified, it was not possible to determine some aspects of the method employed.

We limited our review of impact to only three NCEs, albeit they are the three largest and most longstanding in the UK. We may have missed some grey publications, and it is possible that the recommendations emanating from other NCEs have had a greater impact, though this seems improbable. There are two concerns about the validity and reliability of the data reported in the impact evaluations. First, it is not apparent from the published reports that the questionnaires used in the surveys had been adequately tested before use. Second, most of the studies relied upon the opinion of respondents, usually doctors, rather than on any objective measures, and therefore are vulnerable to hindsight bias.

The final concern is the impossibility of attributing causality using cross-sectional studies, and even the before-and-after designs did not take existing trends and concurrent interventions into account.

Implications

Despite the lack of rigorous evidence of the impact of recommendations emanating from NCEs, clinicians appear to have a strong belief in the value of these enquiries.41 This might be attributed to the non-threatening nature of confidential enquiries in which doctors' clinical practice is discretely reviewed by themselves and their colleagues with no involvement of managers, other healthcare professions or lay people, and all the findings remaining confidential. In addition, most NCEs produce only aggregated data, with no threat of public disclosure (unless required later during legal proceedings), and the data on each patient are destroyed to prevent any later investigation taking place.

The uncertain value of NCEs might not matter if it were not for two factors, validity and cost. The validity of the recommendations based on NCEs is contentious, given the limitations of attributing causality from case series. The lack of comparative data, such as from patients who did not suffer an adverse outcome, and the failure to consider confounding factors and concurrent interventions seriously limits the power and usefulness of NCEs. Despite the design only being appropriate for hypothesis generation, NCEs are frequently used to establish theses and make recommendations.

In addition, the cost of NCEs must be considered. Information on costs is limited, but in England & Wales, the Confidential Enquiry into Maternal And Child Health cost £1.4 m in 2009–2010, and NCEPOD cost £860 k. These represent only the central administrative and analysis costs. In addition, there are the costs of experienced doctors reviewing and reporting on each case. Concern about the lack of information on the cost–benefit ratio of NCEs was recognised by the major funder of NCEs in England and Wales in 2007.42

Recommendations

Improving the safety of healthcare is important. Indeed, it is so important that we need to ensure that the best methods are being employed to understand why adverse outcomes occur and how they might be prevented. Confidential enquiries were devised in the 1950s before computers provided opportunities to collect and handle vast amounts of data on all patients, not just those with adverse outcomes. We can now collect prospective data in national clinical audits that enable assessment of the effectiveness of care and the identification and quantification of factors associated with adverse outcomes. Such databases can also facilitate the evaluation of interventions to prevent such outcomes. Such an approach need not replace or preclude in-depth reviews of the subset of patients who have suffered an adverse outcome, though whether such processes should remain confidential needs further debate. Such an approach in which in-depth enquiries are nested within national audits would facilitate the bringing together of assessments of effectiveness and safety. Whether or not NCEs are incorporated in this way or remain stand-alone, a rigorous evaluation of the cost-effectiveness of the approach to improving the safety of care is needed.

Acknowledgments

AA received a research scholarship from the Alfried Krupp von Bohlen und Halbach-Stiftung, Germany, Nachwuchsförderprogramm ‘Nachwuchsförderung Community Medicine Greifswald.’

References

View Abstract

Footnotes

  • Funding Alfried Krupp von Bohlen und Halbach-Stiftung, Germany, Nachwuchsförderprogramm.

  • Competing interests NB chairs the National Clinical Audit Advisory Group, a non-governmental public body which advises the Department of Health in England.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.