A1 Vertaisarvioitu alkuperäisartikkeli tieteellisessä lehdessä
Empirical investigation of multi-source cross-validation in clinical ECG classification
Tekijät: Leinonen, Tuija; Wong, David; Vasankari, Antti; Wahab, Ali; Nadarajah, Ramesh; Kaisti, Matti; Airola, Antti
Kustantaja: Elsevier
Julkaisuvuosi: 2024
Journal: Computers in Biology and Medicine
Tietokannassa oleva lehden nimi: Computers in Biology and Medicine
Lehden akronyymi: Comput Biol Med
Artikkelin numero: 109271
Vuosikerta: 183
ISSN: 0010-4825
eISSN: 1879-0534
DOI: https://doi.org/10.1016/j.compbiomed.2024.109271
Verkko-osoite: https://doi.org/10.1016/j.compbiomed.2024.109271
Rinnakkaistallenteen osoite: https://research.utu.fi/converis/portal/detail/Publication/458889243
Traditionally, machine learning-based clinical prediction models have been trained and evaluated on patient data from a single source, such as a hospital. Cross-validation methods can be used to estimate the accuracy of such models on new patients originating from the same source, by repeated random splitting of the data. However, such estimates tend to be highly overoptimistic when compared to accuracy obtained from deploying models to sources not represented in the dataset, such as a new hospital. The increasing availability of multi-source medical datasets provides new opportunities for obtaining more comprehensive and realistic evaluations of expected accuracy through source-level cross-validation designs. In this study, we present a systematic empirical evaluation of standard K-fold cross-validation and leave-source-out cross-validation methods in a multi-source setting. We consider the task of electrocardiogram based cardiovascular disease classification, combining and harmonizing the openly available PhysioNet/CinC Challenge 2021 and the Shandong Provincial Hospital datasets for our study. Our results show that K-fold cross-validation, both on single-source and multi-source data, systemically overestimates prediction performance when the end goal is to generalize to new sources. Leave-source-out cross-validation provides more reliable performance estimates, having close to zero bias though larger variability. The evaluation highlights the dangers of obtaining misleading cross-validation results on medical data and demonstrates how these issues can be mitigated when having access to multi-source data.
Ladattava julkaisu This is an electronic reprint of the original article. |  |
Julkaisussa olevat rahoitustiedot:
The authors wish to acknowledge CSC – IT Center for Science, Finland, for generous computational resources. This work has been supported by Research Council of Finland (grants 352893, 358868, 345805, 340140).
