Methods

The hypothesis tested in this study was that, compared with the use of film images, PACS would reduce the number of misdiagnoses by A&E clinicians and consequently the number of patients who had to be recalled to the hospital for reassessment. Thus, the aim of this study was to assess the impact of PACS on the incidence, and consequences, of misdiagnoses in the A&E department at the Hammersmith Hospital. Data were collected over two periods: when conventional film images were used and when PACS was in use. Data were collected on all images reported as positive (that is, abnormality present) by the radiologist but seen as negative (that is, no abnormality present) by the A&E clinician. As a matter of routine audit, a senior A&E clinician compared each relevant radiologist's report of an abnormality with the findings of the A&E Senior House Officer (SHO). If it appeared that a diagnostic error had been made by the A&E clinician, a decision was made about whether treatment should be changed. If a change was thought necessary then the urgency of such a change was defined in terms of whether the patient should be recalled for further treatment. By this method potential false negative findings on radiographic images by A&E clinicians were identified. These cases were classified by the seriousness of the condition and any action recommended. Table 1 gives the classification. The routine audit procedure in operation at the Hammersmith did not monitor all negative radiographic findings by the radiologists. Ideally, false positive findings would also be investigated. However, an investigation of false positive findings could not be undertaken because it was not feasible for the auditing clinicians to check all positive A&E reports with the radiology reports for all patients over two six month periods in addition to their normal workload. Thus, the term “misdiagnosis” is used in this paper to refer to false negative cases only, where the gold standard is taken as the radiologist's report.

Data based on conventional film images were collected from 31 March 1992 to 30 September 1992. The Hammersmith Hospital became “filmless” at the end of March 1996 and the post-PACS data collection period was from 1 April 1996 to 30 September 1996. The same seasonal timing of the “before” and “after” periods was designed to reduce the likelihood that the mix of patients would be different for the two periods, and ensured that the junior A&E staff (who work in A&E for six months starting either at the beginning of February or August) were of the same experience in both studies. By the time of the PACS component of the study, both the A&E and the radiology departments had been using PACS images for more than a year.

Data on the total number of A&E patients radiographed during each study period were obtained from the Radiology Information System (RIS). The details of every radiological examination were routinely recorded on RIS either by the radiographer or by the X-Ray receptionist. It should be noted that some patients had more than one radiological examination and, thus, the total number of examinations was greater than the total number of patients radiographed. Data on the number of patients attending the A&E department during each study period were obtained from the local A&E Register. These data included both new and follow up attenders.

The two rounds of data collection were initially compared in terms of the number of patients presenting with a new problem and the number of follow up attenders. Data were also available on the number of A&E attenders who were radiographed, and the body area of the radiological examination. The data from each round were compared by calculating differences in proportions of patients and 95% confidence intervals around the differences.¹⁷

The overall misdiagnosis rates for the film and PACS periods were then compared. The rates were first calculated using the total number of A&E attenders as the denominator and then recalculated using the number of A&E attenders who were radiographed as the denominator. The misdiagnosis rates for the film and PACS rounds were then compared separately for adults (defined as being 16 years of age or older) and children (defined as being less than 16 years of age). Again, these comparisons were made by calculating 95% confidence intervals around the differences in proportions. Finally, data concerning the distribution of misdiagnoses by body area are reported.

Results

The results for the comparison of film and PACS periods in terms of patient characteristics are given in tables 2 and 3. The mix of patients presenting at the A&E department was different between the two periods, in two important respects. Firstly, a significantly smaller proportion of patients attending in 1996 were follow up patients and, thus, a significantly larger proportion were presenting with a new problem (table 2). Secondly, a significantly larger proportion of patients attending in 1996 received a radiological examination (table 3). Apart from examinations of an upper limb, for all categories of radiological examinations, a larger proportion of patients received an examination in 1996. This increase was particularly marked for examinations of the chest. In 1992, approximately 6% of all A&E attenders received a chest radiograph, whereas in 1996, approximately 12% of attenders had a chest radiograph.

The results for the overall comparison of the film and PACS data collection periods, in terms of misdiagnosis rates, are shown in table 4. During the film period a total of 39 patients were misdiagnosed when film was being used, giving an overall misdiagnosis rate of 1.5% in those patients who were radiographed. The number of patients who were recalled for review (misdiagnosis categories 1 to 3) was 16. During the PACS period a total of 35 patients were misdiagnosed when PACS was being used, giving an overall misdiagnosis rate of 0.66% in those patients who were radiographed. The number of patients in the PACS period who were recalled (misdiagnosis categories 1 to 3) was 20.

The proportion of misdiagnoses among A&E attenders who were radiographed was statistically significantly lower in the period when PACS was being used compared wih the period when film was used. However, the proportion of serious misdiagnoses among A&E attenders who were radiographed was not significantly different between the two periods. The data were analysed separately for adults and children (tables 5 and 6). For adults, there was a significantly lower proportion of misdiagnoses overall when PACS was used, but the rate of serious misdiagnoses, requiring patient recall, was the same for the two periods. For children, the misdiagnosis rates, both overall and for serious misdiagnoses, were the same for the PACS and film periods.

Tables 7 and 8 and figures 1 and 2 show the distribution and severity of misdiagnoses between body areas, for both adults and children. The results indicate that the misdiagnoses tended to relate, principally, to examinations of the upper or lower limb or skull examinations. There are no pronounced differences in the distributions between the film and PACS periods.

Discussion

COMPARISON WITH OTHER STUDIES

The number of misdiagnoses by A&E staff identified in this study was very low, for both film and PACS. However, the rates for both film and PACS images identified here are consistent with those reported in other studies.^1–13 In addition these results compare well with a study of conventional radiographic films by Walsh-Kelly et al¹⁸ who reported that radiograph misinterpretation by emergency department physicians occurs but is unlikely to result in significant morbidity.

This pragmatic study of two periods of normal mixed workload has shown that A&E clinicians on six month rotations were able to use the new technology without detriment to patient care. It is thus in general agreement with the results of the experimental study by Gillard et al¹⁹ who compared the interpretation of digital and film images for skeletal fractures and the acceptance of the technology by A&E staff. In this paper we have not considered the issues of user acceptance and training, but refer readers to other publications about the evaluation of the Hammersmith PACS system, which conclude that the technology has been widely and effectively accepted in the hospital.^20–22

CONFOUNDING FACTORS THAT OCCURRED OVER THE PERIOD OF THE STUDY

The implementation of the full PACS system occurred over a four year time scale and there were several changes other than the implementation of PACS that may have influenced the results of this study. Inevitably there were changes in personnel in the A&E department but the junior doctors were of the same level of experience in both periods of the study. The induction training for the SHOs changed between the periods: at the time of the conventional film study (1992), there was a four day induction training course, which included integrated tuition in the use of imaging. By 1996, training was provided on a two day generic A&E course outside the hospital, with specific training in the use of the PACS system held in house. In addition, by the time of the PACS period there were more “middle grade” doctors available to provide advice during normal working hours, but there had been a reduction in consultant presence. Outside normal working hours, including weekends, there was no change in the grade of doctor available.

For the first data collection period, patients were radiographed in the general radiology department on the floor immediately above A&E. In 1993, a dedicated radiology room for A&E patients was built within the A&E department, which was supervised by a superintendent radiographer with specific responsibility for A&E work. The close proximity of a dedicated A&E radiology room is one possible reason for the increase in requests for radiological examinations during the PACS period. The finding of an increase in the use of radiological examinations can be interpreted either as a change in the characteristics of patients seen in the A&E department or as a change in the behaviour of A&E clinicians for other reasons, such as an increased fear of litigation. There may be a link between the larger proportion of patients receiving a radiological examination in 1996 and the larger proportion of patients presenting with a new problem.

During the film period, the A&E films were “hot reported” during normal working hours but during the PACS study the images were not routinely reported while the patient was in the A&E department. In 1996 the A&E staff made anecdotal comments that the written radiologists' reports took longer to arrive than in the past. These two factors may have caused the A&E staff to rely less on radiologists and instead seek the opinions of other senior clinicians or the A&E radiographers who were located nearer than the radiologists. This process of more frequent consultations may have helped to reduce the number of mistakes that were made by inexperienced junior A&E staff. Balancing this effect, PACS provided the potential for images to be viewed simultaneously in A&E and Radiology. It is possible that this may have led to the A&E doctors being more willing to consult radiologists by phone for an opinion, as this would not necessitate a visit to the radiology department. The consequence of this may have been the observed reduction in the number of images misdiagnosed, although we have no data on the number of telephone consultations.

ACCURACY OF RIS DATA

The accuracy of the RIS information on the numbers of patients radiographed must be treated with caution. It was found in this study that on some occasions a patient's details appeared not to have been entered on RIS even though the patient had been radiographed and the images were available. Thus, the study data on the number of patients radiographed may be an underestimate and so the misdiagnosis rates may be inflated. This is a known inaccuracy in the data that could not be controlled by the researchers. However, there is no reason to believe that such errors would have been made more often in one part of the study than the other.

OTHER RECENT TECHNOLOGICAL DEVELOPMENTS OF PACS

Further technical developments, such as automated audit systems²³ and teleradiology,^24–26 may increase the benefits of PACS. Automated audit can speed up the audit process and allow any necessary changes in treatment to be made earlier. Teleradiology can provide ready access to a radiologist for “hot” or on line reporting of all radiographic images thus reducing the possibility of misdiagnosis of the images by A&E staff.