Beyond the ROC Curve: Activity Monitoring to Evaluate Deep Learning Models in Clinical Settings

Abstract

We evaluated ‘VITALCARE-SEPS’, a deep learning model for sepsis prediction, using the activity monitoring operator characteristics curve with two different scoring algorithms. This evaluation is crucial as the AMOC curve addresses the time-dependent nature of predictions, providing a more nuanced performance assessment than traditional ROC metrics. Our findings demonstrate that the AMOC curve significantly enhances the evaluation of time-series predictions, enabling more accurate and continuous performance monitoring of machine learning models in clinical settings. This approach can improve model deployment and ultimately lead to better patient outcomes in healthcare.