Is it Beneficial to Use Different Thresholds Over Time for Early Prediction Model?

Abstract

In production settings, deep learning models often rely on fixed thresholds. This study investigates whether using varying thresholds over time enhances predictive accuracy and clinical utility, especially for early sepsis prediction. We retrospectively analyzed EMR data from Hallym University Chuncheon Sacred Heart Hospital (2018-2022), excluding patients aged under 18 or without vital signs. Utilizing the AITRICS-VC SEPS deep learning model, which predicts sepsis using six vital signs, eleven lab results and patient information, we examined prediction thresholds at one-hour intervals before sepsis onset. Optimal thresholds for each interval were identified using the Youden index. Net benefit and decision curve analysis compared the performance of time-varying versus global thresholds. Results show interval-specific thresholds yield higher net benefits and increased true positive detections: 456 (0-1 hour), 122 (1-2 hours), 41 (2-3 hours), and 29 (3-4 hours) before sepsis onset. This suggests dynamically adjusting thresholds over time can improve early sepsis detection and patient outcomes.