Research
Triage Tool Inter-rater Reliability: A Comparison of Live Versus Paper Case Scenarios

https://doi.org/10.1016/j.jen.2006.12.016Get rights and content

Background

Published studies of triage scale inter-rater reliability assessment have been conducted mostly using paper case scenarios.

Objective

To determine if this method of inter-rater reliability assessment generated significantly different measures from those generated from live triage cases.

Methods

This was a multicenter, prospective, observational cohort study of a population-based random sample of patients triaged at 2 emergency departments during a period of 4 months. All patients presenting to the emergency department within the study periods were simultaneously and independently triaged using a 5-level triage acuity scale by 2 to 3 research triage nurses blinded to each other's assessment and to the study objective. After 6 months, the same nurses were asked to assign triage scores to paper case scenarios of the same patients that they had each previously triaged.

Results

Each of the 9 research nurses triaged approximately 90 cases. The inter-rater reliabilities as measured by an intraclass correlation coefficient were 0.9 (95% CI = 0.87 to 93) for the live triage assessments and 0.76 (95% CI = 0.73 to 0.79) for the paper case scenarios. The mean triage score assigned to the live cases was significantly less than that assigned to the paper-based cases (3.17; 95% CI = 3.08 to 3.26) (p < 0.001).

Conclusions

There is moderate to high agreement between live cases and paper case scenarios, and the inter-rater reliabilities, although significantly different, are acceptable in both cases. It is impossible to determine which triage setting provides a more accurate triage score but paper case scenarios generally receive lower triage scores than live cases.

Section snippets

Objective

This study was designed to assess the correlation between triage assessments conducted on patients in a live ED setting and those assigned to equivalent paper case scenarios. In addition, we aimed to compare the inter-rater reliability of the tool across both presentation media.

Ethics

This study protocol was submitted to the local Ethics Review Board and found to be exempt from formal ethics review because it was considered a quality assurance study that did not interfere with or alter standard

Results

Triage data were collected (and turned into paper-based cases) for a total of 271 patients. The IRR for the average rating assigned by 3 triage nurses when viewing the cases live was 0.9 (95% CI = 0.89-0.91). When the same cases were reviewed in paper format, the IRR for the average rating assigned by 3 triage nurses was 0.76 (95% CI = 0.74-0.78). The correlation between the research nurses' live triage assessments and the triage assessments they assigned to the paper cases was r = 0.73 (95% CI

Discussion

This study was designed to ensure that selected patients were representative of actual ED case mixes and employed “experienced” triage nurses representative of those performing triage assessments in other hospitals.

The advantages of triage researchers using paper case scenarios are in the savings of cost and time; rather than have a few triage nurses assess a few patients over a long period, multiple cases can be assessed by a large group of observers in a few hours. Although the results show a

Limitations

A limitation of this study is that the research triage nurses were not permitted to directly question patients, because this questioning would interfere with standard patient care. This restriction may have affected their initial triage assessment. However, the level of agreement found between the research nurses' triage assessments and actual triage nurse's assessments was high at 0.71. Also, the scoring of paper case scenarios similarly precluded acquiring additional information other than

Conclusions

Use of paper case scenarios to determine IRR of triage scales is an efficient method that estimates that of live cases. If the results are found to be within an acceptable performance range, further testing of IRR using live cases may be unnecessary.

Andrew Worster is Research Director, Division of Emergency Medicine, McMaster University and Hamilton Health Sciences, Department of Emergency Medicine, Hamilton, Ontario, Canada.

References (9)

There are more references available in the full text version of this article.

Cited by (0)

Andrew Worster is Research Director, Division of Emergency Medicine, McMaster University and Hamilton Health Sciences, Department of Emergency Medicine, Hamilton, Ontario, Canada.

Arlene Sardo is Nurse Practitioner, Hamilton Health Sciences, Department of Emergency Medicine, Hamilton, Ontario, Canada.

Kevin Eva is Epidemiologist, Department of Clinical Epidemiology and Biostatics, McMaster University, Hamilton, Ontario, Canada.

Christopher M.B. Fernandes is Professor of Emergency Medicine, University of Western Ontario, London, Ontario.

Suneel Upadhye is Undergraduate Coordinator, Division of Emergency Medicine, McMaster University and Hamilton Health Sciences, Department of Emergency Medicine, Hamilton, Ontario, Canada.

This study was presented at the 2005 Society of Academic Emergency Medicine Annual Meeting, May 22-25, 2005, New York, NY; the Canadian Association of Emergency Physicians Annual Scientific Meeting, May 30, 2005, Edmonton, Alberta; and the ENA Annual Conference, Nashville, Tenn, September 14-17, 2005.

Funded by a research grant to Dr. Andrew Worster from the Hamilton Emergency Services Network, Hamilton Ontario, Canada.

Authors' note

This study has been published in abstract form as follows:

Sardo A, Worster A, Fernandes CMB, Eva K, Upadhye S. Triage tool inter-rater reliability using live cases vs. paper case scenarios. Acad Emerg Med 2005;12:137.

Worster A, Sardo A, Fernandes CMB, Eva K, Upadhye S. Triage tool inter-rater reliability using live cases vs. paper case scenarios. Can J Emerg Med 2005;7:209.

Worster A, Sardo A, Fernandes C, Eva K, Upadhye S. Triage tool inter-rater reliability using live cases vs. paper case scenarios. J Emerg Nurs 2005;31:426.

View full text