Introduction: Childhood cancer survivors have a higher risk of mental health and adaptive problems compared with their siblings, for example. Assessing the need for psychosocial support is essential for prevention. This project aims to investigate the use of supervised machine learning in the form of text classification in identifying childhood cancer patients needing psychosocial support from nursing notes when at least 1 year had passed from their cancer diagnosis.

Methods: We evaluated three well-known machine learning–based models to recognize patients who had outpatient clinic reservations in the mental health–related care units from free-text nursing notes of 1,672 patients. For model training, the patients were children diagnosed with diabetes mellitus or cancer, while evaluation of the model was done on the patients diagnosed with cancer. A stratified fivefold nested cross-validation was used. We designed this as a binary classification task, with the following labels: no support (0) or psychosocial support (1) was needed. Patients with the latter label were identified by having an outpatient appointment reservation in a mental health–related care unit at least 1 year after their primary diagnosis.

Results: The random forest classification model trained on both cancer and diabetes patients performed best for the cancer patient population in three-times repeated nested cross-validation with 0.798 mean area under the receiver operating characteristics curve and was better with 99% probability (credibility interval −0.2840 to −0.0422) than the neural network–based model using only cancer patients in training when comparing all classifiers pairwise by using the Bayesian correlated t-test.

Conclusions: Using machine learning to predict childhood cancer patients needing psychosocial support was possible using nursing notes with a good area under the receiver operating characteristics curve. The reported experiment indicates that machine learning may assist in identifying patients likely to need mental health–related support later in life.