Research
Evaluating Implementation of the Emergency Severity Index in a Belgian Hospital

https://doi.org/10.1016/j.jen.2014.01.006Get rights and content

Introduction

Triage aims to categorize patients based on their clinical need and the available departmental resources. To accomplish this goal, one needs to ensure that the implemented triage system is reliable and that staff use it correctly. Therefore this study assessed the ability of Belgium nurses to apply the Emergency Severity Index (ESI), version 4, to hypothetical case scenarios after an educational intervention.

Methods

An ESI educational intervention was implemented in accordance with the ESI manual. Using paper case scenarios, nurses’ interrater agreement was assessed by comparing triage nurse ESI levels with the reference answers noted in the implementation manual. Interrater agreement was measured by the percentage of agreement and Cohen’s κ coefficient using different weighting schemes.

Results

Overall, 77.5% of the scenario cases were coded according the ESI guidelines, resulting in a good interrater agreement (κ = 0.72, linear weighted κ = 0.84, quadratic weighted κ = 0.92, and triage-weighted scheme = 0.79). Interrater agreement varied when evaluating each ESI level separately. Undertriage was more common than overtriage. The highest misclassification range (37.8%) occurred in ESI level 2 scenarios, with 99.2% of the misclassifications being undertriaged.

Discussion

Implementation of the ESI into a novel setting guided by a locally developed training program resulted in suboptimal interrater agreement. Existing weighted κ schemes overestimated the interrater agreement between the triage nurse–assigned ESI level and the reference standard. By providing an aggregated measure of agreement, which allows partial agreement, clinically significant misclassification was masked by a misleading “good” interrater agreement.

Section snippets

Design, Setting, and Participants

This cross-sectional observational study was conducted at the emergency department of a 1900-bed tertiary care teaching hospital in Belgium. The emergency department has an annual census of approximately 54 000 patients and a census of 102 to 210 patients per day. Between 2005 and 2008, the patient volume increased by approximately 3814 patients (8%). The ESI was implemented in the emergency department in January 2009. The ED staff had no previous experience with the ESI or triage in general.

Results

The mean score for the survey was 23/30. Overall exact agreement between triage nurses and reference answers was 77.5%. The mistakes made by triage nurses resulted more commonly in undertriage (77.5%) than overtriage. Undertriage ranged from a difference of 1, 2, or 3 levels from the true ESI level; overtriage ranged from a difference of 1 or 2 levels from the true ESI level. The unweighted κ score was 0.72 (95% confidence interval [CI] 0.69-0.74), κlw was 0.84 (95% CI 0.77-0.91), κqw was 0.92

Discussion

Triage is a vital tool in preventing unsafe waiting times for the most vulnerable patients. However, to accomplish this goal, the implemented triage system needs to be valid and reliable and staff need to use the tool correctly. Because triage relies on a nurse’s interpretation of the level specific criteria, interrater variability is problematic but surmountable. In other words, the interrater agreement of a triage system is dependent upon nurses’ knowledge and experience, as well as their

Limitations

When interpreting the results of this study, one needs to be aware of several limitations. First, this study was conducted at a single center with a limited number (N = 52) of triage nurses. Second, because the native language of triage nurses is Dutch, the implementation team had to translate the contents of the implementation manual, and thus potential important nuances could have been lost. Third, triage decision making is a process influenced by multiple factors.12, 13, 14, 15, 16, 17, 18,

Implications for Emergency Nurses

The results of this study suggest that Belgian nurses have difficulties with the interpretation of high acuity level criteria, as well as estimating the number of resources a patient needs to reach a dispositional decision. The ESI manual noted that these decisions are primarily based on nurses’ knowledge and experience. If a triage system is largely based on knowledge and experience, interobserver variation is unavoidable, which implies that sufficient attention must be directed at the

Conclusions

In conclusion, implementation of the ESI into a novel setting guided by a locally developed training program resulted in suboptimal interrater agreement of triage classification. The majority (77.5%) of the assigned triage codes by nurses were in concordance with the true ESI level, and errors were made in all ESI levels, with the highest number of errors in ESI level 2. Further, this study shows that existing weighted κ schemes overestimate the interrater agreement between of the triage

Jochen Bergs is PhD candidate, Patient Safety Research Group, Faculty of Business Economics, Hasselt University, Hasselt, Belgium.

References (27)

  • PC Sprivulis et al.

    The association between hospital overcrowding and mortality among patients admitted via Western Australian emergency departments

    Med J Aust

    (2006)
  • PA Cameron

    Hospital overcrowding: a threat to patient safety?

    Med J Aust

    (2006)
  • RD Mistry et al.

    Urgency classification methods for emergency department visits: do they measure up?

    Pediatr Emerg Care

    (2008)
  • Cited by (18)

    • Educational intervention in triage with the Swedish triage scale RETTS©, with focus on specialist nurse students in ambulance and emergency care – A cross-sectional study

      2022, International Emergency Nursing
      Citation Excerpt :

      The basic educational intervention in this study was brief (4 h) and included recorded lectures and triaging five paper-based scenarios in groups under teacher-led review with focus on basic application rather than deeper reflection on the triage assessment when using RETTS©. In comparison, the implementation of ESI through education consists of a 3-hour theoretical presentation followed by practice sessions on paper-based scenarios, but this path too results in a suboptimal inter-rater agreement of triage classification [48]. Only 4 out of 46 scenarios in this study showed complete agreement, and most scenarios showed some dispersion often across the border between stable and unstable.

    • Reliability and validity of the four-level Chinese emergency triage scale in mainland China: A multicenter assessment

      2020, International Journal of Nursing Studies
      Citation Excerpt :

      Estimates of medical resources available always vary between triage nurses and physicians from different EDs in China. Further, there are reports of the ESI's suboptimal performance, with low accuracy and high variability, outside the United States (Bergs et al., 2014; Jordi et al., 2015). Another factor is the complexities of triage systems in terms of the different presenting complaints (Karjala and Eriksson, 2017).

    • Comparison between Emergency Severity Index plus peak flow meter and Emergency Severity Index in the dyspneic patients with chronic obstructive pulmonary disease: A randomized clinical trial

      2019, Turkish Journal of Emergency Medicine
      Citation Excerpt :

      High-risk criteria cited on ESI level 2 are dependent on nurses’ knowledge of COPD, so this may become a source of discrepancy in triage decisions. In this regard, Bergs et al. showed that the majority of the mistriages that occurred on ESI level 2 were undertriage.17 High-risk criteria in the ESI may be misinterpreted by triage nurses,18 who differ in terms of work experience, knowledge, and skills.

    • Comparison of the Emergency Severity Index versus the Patient Acuity Category Scale in an emergency setting

      2018, International Emergency Nursing
      Citation Excerpt :

      The low number of patients in P4 may indicate that this category may no longer be adequate for use in Singapore. For patient case scenarios, both the ESI and the PACS showed very good quadratic weighted κ values, which were comparable with those in the literature [11,29]. For actual patients, the quadratic weighted κ for the ESI was moderate and similar to previous studies [10,12].

    View all citing articles on Scopus

    Jochen Bergs is PhD candidate, Patient Safety Research Group, Faculty of Business Economics, Hasselt University, Hasselt, Belgium.

    Sandra Verelst is Emergency Physician, Department of Emergency Medicine, Leuven University Hospitals, Leuven, Belgium.

    Jean-Bernard Gillet is Professor, Department of Emergency Medicine, Leuven University Hospitals, Leuven, Belgium.

    Dominique Vandijck is Professor, Patient Safety Research Group, Faculty of Business Economics, Hasselt University, Hasselt, Belgium, and Professor, Department of Public Health and Health Economics, Faculty of Medicine and Health Sciences, Ghent University, Ghent, Belgium.

    View full text