​​​​​​​Purpose To develop and evaluate machine learning and deep learning-based models for automated protocoling of emergency brain MRI scans based on clinical referral text.

Materials and Methods In this single-institution, retrospective study of 1953 emergency brain MRI referrals from January 2016 to January 2019, two neuroradiologists labeled the imaging protocol and use of contrast agent as the reference standard. Three machine learning algorithms (Naïve Bayes, support vector machine, and XGBoost) and two pretrained deep learning models (Finnish BERT and GPT-3.5) were developed to predict the MRI protocol and need for contrast agent. Each model was trained with three datasets (100% of training data, 50% of training data, and 50% + augmented training data). Prediction accuracy was assessed with test set.

Results The GPT-3.5 models trained with 100% of the training data performed best in both tasks, achieving accuracy of 84% (95% CI: 80%-88%) for the correct protocol and 91% (95% CI: 88%-94%) for contrast. BERT had an accuracy of 78% (95% CI: 74%-82%) for the protocol and 89% (95% CI: 86%-92%) for contrast. The best machine learning model in the protocol task was XGBoost (accuracy 78% [95% CI: 73%-82%]), and in the contrast agent task support vector machine and XGBoost (accuracy 88% [95% CI: 84%-91%] for both). The accuracies of two nonneuroradiologists were 80%-83% in the protocol task and 89%-91% in the contrast medium task.

Conclusion Machine learning and deep learning models demonstrated high performance in automatic protocoling emergency brain MRI scans based on text from clinical referrals. Published under a CC BY 4.0 license.