Is it Beneficial to Use Different Thresholds Over Time for Early Prediction Model?

Authors

  • Sungsoo HONG AITRICS, Inc., 218 Teheran-ro, Gangnam-gu, 06221 Seoul, Republic of Korea
  • Hyunwoo CHOO AITRICS, Inc., 218 Teheran-ro, Gangnam-gu, 06221 Seoul, Republic of Korea
  • Kyung Hyun LEE AITRICS, Inc., 218 Teheran-ro, Gangnam-gu, 06221 Seoul, Republic of Korea
  • Sungjun HONG Medical AI Research Center, Research Institute for Future Medicine, Samsung Medical Center, 81 Irwon-ro, Gangnam-gu, 06351 Seoul, Republic of Korea
  • Ki-Byung LEE Division of Pulmonary, Allergy and Critical Care Medicine, Hallym University Chuncheon Sacred Heart Hospital, 77 Sakju-ro, 24253 Chuncheon, Republic of Korea
  • Chang Youl LEE Division of Pulmonary, Allergy and Critical Care Medicine, Hallym University Chuncheon Sacred Heart Hospital, 77 Sakju-ro, 24253 Chuncheon, Republic of Korea

Keywords:

Deep learning, Early prediction, Sepsis, Threshold adjustment, Net benefit

Abstract

In production settings, deep learning models often rely on fixed thresholds. This study investigates whether using varying thresholds over time enhances predictive accuracy and clinical utility, especially for early sepsis prediction. We retrospectively analyzed EMR data from Hallym University Chuncheon Sacred Heart Hospital (2018-2022), excluding patients aged under 18 or without vital signs. Utilizing the AITRICS-VC SEPS deep learning model, which predicts sepsis using six vital signs, eleven lab results and patient information, we examined prediction thresholds at one-hour intervals before sepsis onset. Optimal thresholds for each interval were identified using the Youden index. Net benefit and decision curve analysis compared the performance of time-varying versus global thresholds. Results show interval-specific thresholds yield higher net benefits and increased true positive detections: 456 (0-1 hour), 122 (1-2 hours), 41 (2-3 hours), and 29 (3-4 hours) before sepsis onset. This suggests dynamically adjusting thresholds over time can improve early sepsis detection and patient outcomes.

Downloads

Published

21.11.2024

How to Cite

1.
HONG S, CHOO H, LEE KH, HONG S, LEE K-B, LEE CY. Is it Beneficial to Use Different Thresholds Over Time for Early Prediction Model?. Appl Med Inform [Internet]. 2024 Nov. 21 [cited 2024 Dec. 3];46(Suppl. 2):S5-S8. Available from: https://ami.info.umfcluj.ro/index.php/AMI/article/view/1073