METHODS

A convenience sample of 25 patients was selected from all patients seen in our ED, a tertiary care academic medical centre. The study period was from 15 July to 15 August 2001. Exclusion criteria included age under 18, residence in a correctional facility, inability to give consent, psychiatric complaints, and inability to communicate in English. Patients were also excluded if there was no diagnostic question (for example, trauma such as lacerations, fractures, corneal abrasions, etc).

After consent was obtained, clinical information about the patient visit was prospectively collected. The basic information was recorded by ED examiners (generally a resident or supervised medical student) on a structured data form (T-System, Copyright 2001, www.tsystem.com). To capture any other data that entered into the decision making process, one of the researchers (DVS) accompanied all examiners during all contacts with each patient. All of these contacts were audiotaped and relevant clinical data from tape transcriptions were abstracted and entered into a structured data form. All other entries in the medical record including dictated notes, laboratory results, etc, were reviewed after the visit and any additional data gleaned were added to the case abstract. Only data available in the ED were included in the case abstracts. Thus, the ED physician EKG and radiograph interpretations were used in the case abstract. One of the researchers (MG) was available to assist in interpreting terminology or other data and reviewed all of the cases. The final ED discharge diagnoses from the medical records are listed in table 1⇓.

All of the clinical information including history, physical findings, laboratory findings, radiographic findings, EKG findings, etc, was entered into QMR and Iliad. The differential diagnosis generated by the computer programs was compared with the final diagnosis of the ED physicians, all of whom were (and are) academic faculty in emergency medicine. The performance of the computer programs was determined by comparing the number of cases where the attending staffs’ diagnosis was listed on the programs’ differential list. As the purpose of this study was to determine how well QMR and Iliad perform using only the information collected in the ED, any information obtained while the patient was an inpatient or at subsequent outpatient visits was not considered. Likewise, the “final” diagnosis made in the hospital or at subsequent visits was not considered. This design was chosen because the purpose of this study was not to determine the accuracy of ED diagnoses but rather so see how QMR and Iliad performed compared with experienced ED physicians. The Human Subjects committee at our institution approved this protocol.

RESULTS

The results are summarised in table 2⇓. Each case took from 20 to 40 minutes to input.

DISCUSSION

When defining success as having the ED physicians’ diagnosis within the top five generated by QMR and Iliad, the programs were successful in about one third of the cases. Overall, the ED physicians’ diagnosis appeared somewhere in differential diagnosis with about the same frequency that has been found in other studies (around 50%–70%).⁶ Thus, while these programs may be helpful in a general sense, applying them as an arbitrator in any particular case may be problematic. The considerable length of the differential produced by the programs, often greater than 30 diagnoses, and the length of time it takes to input a case may hinder their usefulness. The strength of these programs is their ability to expand the differential diagnosis and make suggestions about further testing and evaluation.^10,11 This can be done by entering a few key findings. Further study of decision support software for this purpose in the ED is warranted.

Several limitations in the design of these programs became clear during this study. The type of information that can be entered into the programs is limited. Iliad and QMR do not take into account the drugs that a patient may be taking. The ability to input the duration of signs and symptoms is also limited, especially with QMR, and the programs are unable to account for the sequence of symptom development. Other findings are simply not in the programs’ vocabulary despite trying multiple synonyms and therefore cannot be entered as part of case. Further development of these programs may mitigate these problems.

The main limitation of this study is the reliance on the ED physicians’ diagnosis as the “criterion standard”. The final diagnosis after hospitalisation or subsequent outpatient visits may or may not differ from the ED diagnosis. However, this is after additional information has been collected and additional time has elapsed. We concentrated on how these programs perform given the limited information collected in the ED. All of the physicians involved in the study were academic emergency physicians at a major teaching university. So, even though the final, post-hospitalisation, diagnosis may vary from that made in the ED, this study reflects the best available diagnosis in the ED.

In conclusion, diagnostic decision support software has the same success in finding the “correct” diagnosis in the ED as has been found in other clinical settings. The accuracy is not sufficiently high to permit the use of these programs as an arbiter in any individual case. However, they can be used to broaden the differential diagnosis.