Article Text

Download PDFPDF
Grand rounds in methodology: key considerations for implementing machine learning solutions in quality improvement initiatives
  1. Amol A Verma1,2,3,4,
  2. Patricia Trbovich2,5,6,
  3. Muhammad Mamdani1,2,4,
  4. Kaveh G Shojania4,7
  1. 1 Unity Health Toronto, Toronto, Ontario, Canada
  2. 2 Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada
  3. 3 Laboratory Medicine and Pathobiology, University of Toronto, Toronto, ON, Canada
  4. 4 Medicine, University of Toronto Faculty of Medicine, Toronto, Ontario, Canada
  5. 5 Centre for Quality Improvement and Patient Safety, Department of Medicine, University of Toronto, Toronto, ON, Canada
  6. 6 North York General Hospital, Toronto, ON, Canada
  7. 7 Sunnybrook Health Sciences Centre, Toronto, ON, Canada
  1. Correspondence to Dr Amol A Verma, Unity Health Toronto, Toronto, Ontario, Canada; amol.verma{at}


Machine learning (ML) solutions are increasingly entering healthcare. They are complex, sociotechnical systems that include data inputs, ML models, technical infrastructure and human interactions. They have promise for improving care across a wide range of clinical applications but if poorly implemented, they may disrupt clinical workflows, exacerbate inequities in care and harm patients. Many aspects of ML solutions are similar to other digital technologies, which have well-established approaches to implementation. However, ML applications present distinct implementation challenges, given that their predictions are often complex and difficult to understand, they can be influenced by biases in the data sets used to develop them, and their impacts on human behaviour are poorly understood. This manuscript summarises the current state of knowledge about implementing ML solutions in clinical care and offers practical guidance for implementation. We propose three overarching questions for potential users to consider when deploying ML solutions in clinical care: (1) Is a clinical or operational problem likely to be addressed by an ML solution? (2) How can an ML solution be evaluated to determine its readiness for deployment? (3) How can an ML solution be deployed and maintained optimally? The Quality Improvement community has an essential role to play in ensuring that ML solutions are translated into clinical practice safely, effectively, and ethically.

  • healthcare quality improvement
  • implementation science
  • information technology
  • quality improvement methodologies

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.


  • Twitter @AmolAVerma

  • Contributors AAV drafted the initial version of the manuscript and is the guarantor of this work. All authors revised the manuscript for important intellectual content.

  • Funding This study was funded by University of Toronto (Temerty Professorship in AI Research and Education).

  • Competing interests AAV and MM are co-inventors of CHARTwatch, an artificial intelligence early warning system for patient deterioration and have the potential to acquire minority interests in a start-up company, Signal1.

  • Provenance and peer review Commissioned; externally peer reviewed.