Can a Checklist Reduce SOS Errors in Chest Radiography?

doi:10.1016/j.acra.2005.11.032

Academic Radiology

Volume 13, Issue 3, March 2006, Pages 296-304

https://doi.org/10.1016/j.acra.2005.11.032 Get rights and content

Rationale and Objectives

A previous study demonstrated unexpected protection from satisfaction of search (SOS) effects when observers verbalized the focus of their attention during visual search and interpretation of chest radiographs. We suggested that protection from SOS might have occurred if each observer developed an informal checklist to help generate the verbal descriptions. The objective of this study is to determine whether a formal checklist reduces SOS effects in chest radiology.

Materials and Methods

Fifty-seven chest radiographs, half of which demonstrated diverse, native abnormalities were read twice by 20 observers, once with and once without the addition of a simulated pulmonary nodule. Area under the receiver operating characteristic (ROC) curve for detecting the native abnormalities was estimated for each observer in each treatment condition using the contaminated binormal ROC model. Radiologists in the current experiment used a checklist during the interpretation, rather than describing their visual search. Results were compared with those of the verbalization study, which used the same set of radiographs.

Results

Although no SOS effect was found when the checklist was used, ROC performance was, on average, much poorer with the checklist than when ongoing search was reported verbally (0.68 versus 0.75, F(1,37) = 17.26, P < .001).

Conclusions

Our results indicate that the recommendation to use a self-prompting checklist to counteract SOS is not warranted. The relative superiority of verbalizing search over using an imposed checklist may be based on the consistency of each of these interventions with the observer’s internal strategy for searching radiographs.

Section snippets

Experimental Conditions

To test whether a checklist could alter the satisfaction of search effect, we used the same two conditions that were used in previous SOS demonstrations: presentation of each chest radiograph with and without a simulated pulmonary nodule. The detection accuracy for native, subtle lesions was compared with that for those same lesions when a simulated pulmonary nodule was added photographically to the radiograph. Thus the background anatomy and actual lesions were perfectly matched for the two

Results

Figure 2 presents a graphic representation of the results of the current experiment and those of earlier experiments. In the checklist experiment, the paired t-test on contaminated binormal areas found no significant difference without and with added nodules (0.67 versus 0.68, t(19) = −0.56, P = .58); the checklist eliminated the SOS effect. The SOS experiment (3) previously reported a t-test on the data of the 19 observers, demonstrating an SOS effect on area. We repeated this analysis just on

Discussion

Describing visual search during radiologic interpretation in the previous experiment was not designed to prevent the SOS effect (5); that result was entirely unexpected. The verbal descriptions were collected to help find the causes of SOS. The authors explained the protection from SOS by proposing that observers formulated an internal checklist on the fly to help them with reporting their search behavior. The results of the current experiment show that this explanation cannot be correct. An

References (34)

K.S. Berbaum et al.
Proper ROC analysis and joint ROC analysis of the satisfaction of search effect in chest radiography
Acad Radiol
(2000)
K.S. Berbaum et al.
The role of faulty decision making in the satisfaction of search effect in chest radiology
Acad Radiol
(2000)
D.D. Dorfman et al.
Proper receiver operating characteristic analysisThe bigamma model
Acad Radiol
(1997)
C.E. Metz et al.
“Proper” binormal ROC curvesTheory and maximum-likelihood estimation
J Math Psychol
(1999)
D.D. Dorfman et al.
A contaminated binormal model for ROC data—part II. A formal model
Acad Radiol
(2000)
K.S. Berbaum et al.
Satisfaction of search in diagnostic radiology
Invest Radiol
(1990)
K.S. Berbaum et al.
Time course of satisfaction of search
Invest Radiol
(1991)
S. Samuel et al.
Mechanism of satisfaction of searchEye position recordings in the reading of chest radiographs
Radiology
(1995)
C.F. Nodine et al.
Using eye movements to study visual search and to improve tumor detection
RadioGraphics
(1987)
K.S. Berbaum et al.
The influence of clinical history on visual search with single and multiple abnormalities
Invest Radiol
(1993)

A.G. Gale et al.

The utility of scanning strategies in radiology

D.P. Carmody et al.

Comparison scans while reading chest imagesTaught, but not practiced

Invest Radiol

(1984)

H.L. Kundel

Visual sampling and estimates of the location of information on chest films

Invest Radiol

(1974)

D.P. Chakraborty et al.

Observer studies involving detection and localizationModeling, analysis, and validation

Med Phys

(2004)

H.E. Rockette et al.

Nonparametric estimation of degenerate ROC data sets used for comparison of imaging systems

Invest Radiol

(1990)

W.J. Dixon

R.E. Kirk

Cited by (44)

Checklists for Interpreting Chest Radiographs: A Scoping Review
2023, Journal of Radiology Nursing
Checklists reduce time to comprehensive radiographic reports, improve quality and consistency of abnormality detection on chest radiographs.
The aim of this review was to examine and survey the scope of published research on checklists for the interpretation of chest radiographs.
We conducted a search of CINAHL, Scopus, Medline Ovid, Web of Science, ProQuest, and gray literature using search terms: chest radiographs, checklist, and image interpretation. Data were extracted from 16 articles. Data was analyzed numerically and thematically.
The selected studies were conducted in the United States (37.5%), the United Kingdom (25%), Australia (12%), South Africa (12%), Turkey (6%), and Israel (6%). The codes were grouped into five categories related to the use of checklists, in chest interpretation.
In the selected studies, reports showed that there was no checklist for chest interpretation in South Africa and no evidence supporting checklists as an interprofessional communication tool for chest interpretation. The authors of this study recommended a chest interpretation checklist should be developed for use by health care professionals practicing in resource-limited settings where radiologists are not on site.
Investigating the impact of cognitive biases in radiologists’ image interpretation: A scoping review
2023, European Journal of Radiology
Image interpretation is a fundamental aspect of radiology. The treatment and management of patients relies on accurate and timely imaging diagnosis. However, errors in radiological reports can negatively impact on patient health outcomes. These misdiagnoses can be caused by several different errors, but cognitive biases account for 74 % of all image interpretation errors. There are many biases that can impact on a radiologist’s perception and cognitive processes. Several recent narrative reviews have discussed these cognitive biases and have offered possible strategies to mitigate their effects. However, these strategies remain untested. Therefore, the purpose of this scoping review is to evaluate the current knowledge on the extent that cognitive biases impact on medical image interpretation.
Scopus and Medline Databases were searched using relevant keywords to identify papers published between 2012 and 2022. A subsequent hand search of the narrative reviews was also performed. All studies collected were screened and assessed against the inclusion and exclusion criteria.
Twenty-four publications were included and categorised into five main themes: satisfaction of search, availability bias, hindsight bias, framing bias and other biases. From these studies, there were mixed results regarding the impact of cognitive biases, highlighting the need for further investigation in this area. Moreover, the limited and untested debiasing methods offered by a minority of the publications and narrative reviews also suggests the need for further research. The potential of role of artificial intelligence is also highlighted to further assist radiologists in identifying and mitigating these cognitive biases.
Cognitive biases can impact radiologists’ image interpretation, however the effectiveness of debiasing strategies remain largely untested.
Biases on visual warnings during multiple target visual searches on mine monitoring interface
2023, Displays
With the continuous development of human–computer interaction, some industrial monitoring systems have gradually evolved from a single monitoring device to a comprehensive monitoring system platform, and multi-objective visual search is particularly important in the field of industrial monitoring. This study investigates the effect of conceptual and perceptual biases on similarity bias in multiple target visual searches and proposes relevant design strategies that can be applied to warning interfaces. The results of two experiments show that the effect of the similarity bias is due to the combined effects of conceptual and perceptual similarities, and on this basis, we propose a design strategy for the interface of a gas extraction monitoring system used in mines in a third experiment. An existing interface to monitor gas extraction and a proposed interface with modifications made to the existing interface are evaluated. The results confirm that the modified interface effectively improves visual search performance during multiple target visual searches. The research findings are important for reducing cognitive decision errors during related monitoring work and ensuring the performance of the mine gas safety monitoring system interfaces. The findings can also guide the designing of visual warning elements of different targets for other monitoring interfaces.
Reducing Errors Resulting From Commonly Missed Chest Radiography Findings
2023, Chest
Chest radiography (CXR), the most frequently performed imaging examination, is vulnerable to interpretation errors resulting from commonly missed findings. Methods to reduce these errors are presented. A practical approach using a systematic and comprehensive visual search strategy is described. The use of a checklist for quality control in the interpretation of CXR images is proposed to avoid overlooking commonly missed findings of clinical importance. Artificial intelligence is among the emerging and promising methods to enhance detection of CXR abnormalities. Despite their potential adverse consequences, errors offer opportunities for continued education and quality improvements in patient care, if managed within a just, supportive culture.
Accuracy of radiographers in Fiji in interpreting adult chest X-ray images
2022, Journal of Medical Imaging and Radiation Sciences
Citation Excerpt :
Kim and Mansfield [15] attribute this as errors of location whereby important musculoskeletal findings are overlooked at chest radiograph interpretation because it is outside the area of interest on an image. This type of error, according to the authors also falls into the “satisfaction of search” category which was stated as the second most common errors encountered during image interpretation [15] and is a well-recognised phenomenon in chest image interpretation [15,16,34,35]. In the current study, each image is presumed to contain one pathology according to the consultant radiologist's report but since the images were not validated by the arbiter(s), this error cannot be completely ruled out amongst the participants of this study.
The idea of radiographer image interpretation has not been fully explored in Fiji despite the great shortage of radiologists in the country. This is a feasibility study of radiographer image interpretation aimed at assessing the accuracy of radiographers in interpreting adult chest X-ray images at the Colonial War Memorial Hospital (CWMH) in Fiji.
Forty PA chest X-ray images were interpreted in terms of correctly commenting on the presence/ absence of pathology, the name of the pathology, and the location of the pathology on the data sheets by the CWMH diagnostic radiographers (n = 14). The radiographers were grouped according to their years of work experience (≤ 5 vs > 5) and the Mann Whitney U test on a two-tailed p-value of 0.05 was used to compare this grouping.
The data analysis was conducted using the Statistical Package for Social Sciences (SPSS) v.25 and Microsoft Excel. The diagnostic performance of all radiographers in terms of triaging normal and abnormal in the images shows sensitivity ranging from 71.4 to 100%, with specificity ranging from 47.4 to 100%. The mean sensitivity, specificity, and the overall accuracy of radiographers in triaging normal and abnormal on the images were 89.5%, 72.9%, and 81.6%, respectively. The mean accuracy in naming the pathology was 33.6% and the location sensitivity was 45.7%. There was no statistically significant difference in results between the radiographers’ years of experience.
Without any formal qualification, training, and practice, the current results suggest that the cohort of radiographers can perform normal/abnormal triage of CXRs within a test setting. With a significant reduction in the radiographers’ accuracy in terms of naming and locating the abnormality, the study results do not support image interpretation by the radiographers at this stage.
This feasibility study provides baseline information about the accuracy of image interpretation by diagnostic radiographers at CWMH and provides a platform for further research in image interpretation in Fiji.
L'idée de l'interprétation d'images par des radiographes n'a pas été pleinement explorée aux Fidji malgré la grande pénurie de radiologues dans le pays. Il s'agit d'une étude de faisabilité de l'interprétation d'images par un radiographe visant à évaluer la précision des radiographes dans l'interprétation des radiographies pulmonaires d'adultes au Colonial War Memorial Hospital (CWMH) de Fidji.
Quarante radiographies pulmonaires ont été interprétées en termes de commentaires corrects sur la présence/absence de pathologie, le nom de la pathologie et l'emplacement de la pathologie sur les feuilles de données par les radiographes diagnostiques du CWMH (n=14). Les radiographes ont été regroupés en fonction de leurs années d'expérience professionnelle (≤ 5 c. > 5) et le test U de Mann Whitney sur une valeur p bilatérale de 0,05 a été utilisé pour comparer ce regroupement.
L'analyse des données a été réalisée à l'aide du Statistical Package for Social Sciences (SPSS) v.25 et de Microsoft Excel. La performance diagnostique de tous les radiographes en termes de triage des images normales et anormales montre une sensibilité allant de 71,4 % à 100 %, avec une spécificité allant de 47,4 % à 100 %. La sensibilité moyenne, la spécificité et la précision globale des radiographes dans le triage du normal et de l'anormal sur les images étaient respectivement de 89,5%, 27,1% et 81,6%. La précision moyenne pour nommer la pathologie était de 33,6 % et la sensibilité de localisation était de 45,7 %. Le nombre d’années d’expérience des radiographes n’a pas produit de différence statistiquement significative dans les résultats.
Sans qualification, formation et pratique formelles, les ré sultats actuels suggèrent que la cohorte de radiographes peut effectuer un triage normal/anormal des radiographies dans un contexte de test. Avec une réduction significative de la précision des radiographes en termes de désignation et de localisation de l'anomalie, les résultats de l'étude ne soutiennent pas l'interprétation des images par les radiographes à ce stade.
Cette étude de faisabilité fournit des informations de base sur la précision de l'interprétation des images par les radiographes de diagnostic au CWMH et fournit une plate-forme pour des recherches ultérieures sur l'interprétation des images aux Fidji.
The influence of a vocalized checklist on detection of multiple abnormalities in chest radiography
2016, Academic Radiology
Although a checklist has been recommended for preventing satisfaction of search (SOS) errors, a previous research study did not demonstrate that benefit. However, observers in that study had to turn away from the image display to use the checklist. The current study tested a vocalized checklist to avoid this constraint.
A total of 64 chest computed radiographs, half containing various “test” abnormalities, were read twice by 20 radiologists, once with and once without the addition of a simulated pulmonary nodule. Readers used a vocalized checklist-directing search. Receiver operating characteristic (ROC) detection accuracy and decision thresholds were analyzed to study the effects of adding the nodule on detecting the test abnormalities.
Adding nodules induced a substantial reluctance to report the other abnormalities (P < 0.001), as had been the case in the most recent study of the SOS effect in radiography.
The vocalized checklist did not reduce nor eliminate the SOS effect on readiness to report further abnormalities. Although useful for organizing search and reporting, particularly among students, a vocalized checklist does not prevent SOS effects.

View all citing articles on Scopus

¹: Supported by USPHS Grants R01 EB/CA00145, R01 EB/CA00863, R01 CA42453, and R01 CA62362 from the National Cancer Institute, Bethesda, MD.

View full text

Original investigationCan a Checklist Reduce SOS Errors in Chest Radiography?1

Rationale and Objectives

Materials and Methods

Results

Conclusions

Section snippets

Experimental Conditions

Results

Discussion

Acad Radiol

Acad Radiol

Acad Radiol

J Math Psychol

Acad Radiol

Satisfaction of search in diagnostic radiology

Invest Radiol

Time course of satisfaction of search

Invest Radiol

Mechanism of satisfaction of searchEye position recordings in the reading of chest radiographs

Radiology

Using eye movements to study visual search and to improve tumor detection

RadioGraphics

The influence of clinical history on visual search with single and multiple abnormalities

Invest Radiol

The utility of scanning strategies in radiology

Comparison scans while reading chest imagesTaught, but not practiced

Invest Radiol

Visual sampling and estimates of the location of information on chest films

Invest Radiol

Observer studies involving detection and localizationModeling, analysis, and validation

Med Phys

Nonparametric estimation of degenerate ROC data sets used for comparison of imaging systems

Invest Radiol

Original investigation
Can a Checklist Reduce SOS Errors in Chest Radiography?1