Objective Delirium is a severe condition, often underreported and linked to adverse outcomes such as increased mortality and prolonged hospitalization. Despite its significance, delirium prediction is often hindered by underreporting and inconsistent labeling, highlighting the need for models trained on reliably labeled data (ground truth). This review examines (i) practices for determining labels in delirium prediction models and (ii) how study designs affect label quality, aiming to identify key considerations for improving model reliability.

Materials and Methods A search of Cochrane, PubMed, and IEEE identified 120 studies that met the inclusion criteria.

Results To establish the ground truth, 40.8% of studies used routine data, while 42.5% used primary data. The Confusion Assessment Method (CAM) was the most widely used assessment tool (60. 0%). Label and data leakage occurred in 35.0% of studies. High Risk of Bias (RoB) was a recurring issue, with 31.7% of studies lacking sufficient reporting and 36.7% showing inadequate outcome determination. Studies using primary data had lower RoB, whereas those with unclear label sources displayed higher RoB.

Discussion Our findings underscore the importance of careful planning in determining the ground truth frequently neglected in existing studies. To address these challenges, we provide a decision support flowchart to guide the development of more accurate and reliable prediction models.

Conclusion This review uncovers significant variability in labeling methods and discusses how this may affect delirium prediction model reliability. Highlighting the importance of addressing underreporting bias and providing guidance for developing more robust models.

Delirium, a serious condition causing acute confusion in hospitalized patients, is linked to worse outcomes, including longer hospital stays, higher costs, and increased mortality. However, it is often underreported, making it difficult for artificial intelligence (AI) models to predict accurately. When hospitals fail to document delirium cases, AI models may only detect severe cases already flagged by clinicians. Our review of 120 studies examined why prediction models struggle with reliability. Nearly half relied on routine hospital records prone to underreporting, while others used data collected specifically for model development. The Confusion Assessment Method was the most common tool, but over a third of studies had errors, such as including post-outcome data in predictions or using unclear labels, which can falsely inflate accuracy. Models using direct assessments performed better, emphasizing the need for high-quality data. To address these issues, we developed a step-by-step guide to help researchers and clinicians build fairer, more reliable models. This tool promotes careful planning to reduce bias and improve detection of overlooked cases. By improving data quality, healthcare teams can create AI-driven prediction tools that better identify delirium early, ultimately reducing complications and improving patient outcomes.