Continuous loneliness assessment is important for mental health monitoring and early intervention. Existing multimodal methods primarily rely on supervised machine learning models trained on population-level patterns, which require substantial amounts of labelled data and may fail to capture individual baseline differences, leading to underestimation of high-risk states. Our study proposes a Multimodal-to-Semantics Large Language Model (M2S-LLM) framework for assessing loneliness levels. The M2S-LLM transforms multimodal physiological and behavioural signals from wearable devices and smartphones into baseline-relative semantic representations, and incorporates limited historical Ecological Momentary Assessment labels as a lightweight local knowledge base for LLM-based contextual reasoning. Experimental results on a real-world dataset show that the M2S-LLM substantially reduces loneliness-level underestimation and yields a more balanced error distribution in the high-loneliness range. The results indicate its potential for personalised and risk-sensitive loneliness assessment under limited-label conditions.