A general framework for considering clinical artificial intelligence (AI) quality and safety issues in medicine
| Issue | Summary | Example |
| Short term | ||
| Distributional shift | A mismatch between the data or environment the system is trained on and that used in operation, due to bias in the training set, change over time, or use of the system in a different population, may result in an erroneous ‘out of sample’ prediction. | The accuracy of a system which predicts impending acute kidney injury based on other health records data, became less accurate over time as disease patterns changed.40 |
| Insensitivity to impact | A system makes predictions that fail to take into account the impact of false positive or false negative predictions within the clinical context of use. | An unsafe diagnostic system is trained to be maximally accurate by correctly diagnosing benign lesions at the expense of occasionally missing malignancy.6 |
| Black box decision making | A system’s predictions are not open to inspection or interpretation and can only be judged as correct based on the final outcome. | A X-Ray analysis AI system could be inaccurate in certain scenarios because of a problem with training data, but as a black box this is not possible to predict and will only become apparent after prolonged use.9 |
| Unsafe failure mode | A system produces a prediction when it has no confidence in the prediction accuracy, or when it has insufficient information to make the prediction. | An unsafe AI decision support system may predict a low risk of a disease when some relevant data is missing. Without any information about the prediction confidence, a clinician may not realise how untrustworthy this prediction is.46 |
| Medium term | ||
| Automation complacency | A system’s predictions are given more weight than they deserve as the system is seen as infallible or confirming initial assumptions. | The busy clinician ceases to consider alternatives when a usually predictable AI system agrees with their diagnosis.48 |
| Reinforcement of outmoded practice | A system is trained on historical data which reinforces existing practice, and cannot adapt to new developments or sudden changes in policy | A drug is withdrawn due to safety concerns but the AI decision support system cannot adapt as it has no historical data on the alternative. |
| Self-fulfilling prediction | Implementation of a system indirectly reinforces the outcome it is designed to detect. | A system trained on outcome data, predicts that certain cancer patients have a poor prognosis. This results in them having palliative rather than curative treatment, reinforcing the learnt behaviour. |
| Long term | ||
| Negative side effects | System learns to perform a narrow function that fails to take account of some wider context creating a dangerous unintended consequence. | An autonomous ventilator derives a ventilation strategy that successfully maintains short term oxygenation at the expense of long-term lung damage.34 |
| Reward hacking | A proxy for the intended goal is used as a ‘reward’ and a continuously learning system finds an unexpected way to achieve the reward without fulfilling the intended goal. | An autonomous heparin infusion finds a way to control activated partial thromboplastin time (aPTT) at the time of measurement without achieving long-term control.23 |
| Unsafe exploration | An actively learning system begins to learn new strategies by testing boundary conditions in an unsafe way. | A continuously learning autonomous heparin infusion starts using dangerously large bolus doses to achieve rapid aPTT control.23 |
| Unscalable oversight | A system requires a degree of monitoring that becomes prohibitively time consuming to provide. | An autonomous subcutaneous insulin pump requires the patient to provide exhaustive detail of everything they have eaten before it can adjust the insulin regime.33 |