Beyond the ROC Curve: Activity Monitoring to Evaluate Deep Learning Models in Clinical Settings

Authors

  • Hyunwoo CHOO AITRICS Inc., 218 Teheran-ro, Gangnam-gu, 06221Seoul, Republic of Korea
  • Kyung Hyun LEE AITRICS Inc., 218 Teheran-ro, Gangnam-gu, 06221Seoul, Republic of Korea
  • Sungsoo HONG AITRICS Inc., 218 Teheran-ro, Gangnam-gu, 06221Seoul, Republic of Korea
  • Sungjun HONG Medical AI Research Center, Research Institute for Future Medicine, Samsung Medical Center, 81 Irwon-ro, Gangnam-gu, 06351 Seoul, Republic of Korea
  • Ki-Byung LEE Division of Pulmonary, Allergy and Critical Care Medicine, Hallym University Chuncheon Sacred Heart Hospital, 77 Sakju-ro, 24253 Chuncheon, Republic of Korea
  • Chang Youl LEE Division of Pulmonary, Allergy and Critical Care Medicine, Hallym University Chuncheon Sacred Heart Hospital, 77 Sakju-ro, 24253 Chuncheon, Republic of Korea

Keywords:

Deep learning, Sepsis, Activity monitoring, ROC (receiver operating characteristic) curve

Abstract

We evaluated ‘VITALCARE-SEPS’, a deep learning model for sepsis prediction, using the activity monitoring operator characteristics curve with two different scoring algorithms. This evaluation is crucial as the AMOC curve addresses the time-dependent nature of predictions, providing a more nuanced performance assessment than traditional ROC metrics. Our findings demonstrate that the AMOC curve significantly enhances the evaluation of time-series predictions, enabling more accurate and continuous performance monitoring of machine learning models in clinical settings. This approach can improve model deployment and ultimately lead to better patient outcomes in healthcare.

Downloads

Published

21.11.2024

How to Cite

1.
CHOO H, LEE KH, HONG S, HONG S, LEE K-B, LEE CY. Beyond the ROC Curve: Activity Monitoring to Evaluate Deep Learning Models in Clinical Settings. Appl Med Inform [Internet]. 2024 Nov. 21 [cited 2024 Dec. 3];46(Suppl. 2):S9-S12. Available from: https://ami.info.umfcluj.ro/index.php/AMI/article/view/1074