Background: Prostate cancer (PCa) is the most common malignancy among men in the Western world and a leading cause of cancer-related mortality. Machine learning (ML) models leveraging radiomic features from Magnetic Resonance Imaging (MRI) have shown promise in improving diagnostic accuracy. However, a crucial question remains: Can radiomics-based models maintain their performance across different MRI scanners, or do vendor-specific variations undermine their reliability? This study systematically evaluates the reproducibility and robustness of ML models trained on radiomic features extracted using Pyradiomics and MRCradiomics across MRI scanners from different manufacturers. 

Methods: We analyzed imaging data from 637 men with clinical suspicion of PCa, obtained from multiple clinical trials. Axial T2-weighted MRI scans were acquired using Siemens MAGNETOM Verio 3T and Philips Ingenia 3T scanners. Radiomic features were extracted using Pyradiomics and MRCradiomics, resulting in 2693 features. Feature selection via Maximum Relevance Minimum Redundancy (MRMR) reduced this set to 14 highly predictive features. Two ML models “Support Vector Machine (SVM) and Random Forest (RF)” were trained and evaluated on distinct training, validation, and test datasets, with performance assessed using Area Under the Curve (AUC) metrics. 

Results: The SVM model, trained on combined Pyradiomics and MRCradiomics features, achieved an AUC of 0.74 on the Multi-Improd dataset, yet its performance dropped drastically to 0.35 on the Philips test set, indicating poor cross-vendor reproducibility. Similarly, the Random Forest model performed well on the Multi-Improd dataset (AUC = 0.73) but declined to 0.60 on the Philips set. Interestingly, models trained solely on Pyradiomics features demonstrated greater robustness, with the Random Forest model achieving an AUC of 0.78 on the Philips test set. 

Conclusions: While radiomics-based ML models show promise for PCa detection, their generalizability across MRI scanners is far from guaranteed. Performance disparities across vendors highlight the critical need for standardized radiomic feature extraction pipelines to ensure model reliability in real-world clinical applications. Our findings suggest that some radiomics-based models can survive across MRI scanners, but only under carefully controlled conditions—reinforcing the importance of cross-vendor validation in AI-driven diagnostic tools.