A4 Refereed article in a conference publication
Risk Detection in E-commerce with LLMs: Annotation Challenges and Lessons from Real-World Business News
Authors: Davoodi, Laleh; Salimi, Sima; Ginter, Filip; Lorentz, Harri
Editors: Achilleos, Achilleas; Forti, Stefano; Papadopoulos, George Angelos; Pappas, Ilias
Conference name: IFIP Conference on e-Business, eServices, and e-Society
Publisher: Springer Science and Business Media Deutschland GmbH
Publication year: 2025
Journal: Lecture Notes in Computer Science
Book title : Pervasive Digital Services for People’s Well-Being, Inclusion and Sustainable Development : 24th IFIP WG 6.11 Conference on e-Business, e-Services and e-Society, I3E 2025, Limassol, Cyprus, September 9–11, 2025, Proceedings
Volume: 16079
First page : 146
Last page: 160
ISBN: 978-3-032-06163-8
eISBN: 978-3-032-06164-5
ISSN: 0302-9743
eISSN: 1611-3349
DOI: https://doi.org/10.1007/978-3-032-06164-5_11
Publication's open availability at the time of reporting: No Open Access
Publication channel's open availability : No Open Access publication channel
Web address : https://link.springer.com/chapter/10.1007/978-3-032-06164-5_11
The growing complexity of e-commerce supply chains has amplified the need for effective risk monitoring systems. While Large Language Models (LLMs) have demonstrated potential in various domains, their application to real-world risk detection in e-commerce remains underexplored. This study introduces a novel, manually annotated dataset of 121 business news articles covering five major e-commerce-related steel companies, ArcelorMittal, Tata Steel, POSCO, NLMK, and ThyssenKrupp, annotated using the Cambridge Risk Taxonomy. We evaluate the performance of two advanced LLMs in detecting and classifying risks across multiple categories using few-shot prompting and semantic similarity-based example selection. Our results show that LLMs can approximate human annotation with moderate micro F1-scores and high coverage, though challenges remain in recognizing complex Geopolitical risks and avoiding overgeneralization. The findings provide actionable insights into the potential and limitations of LLMs for automated, domain-aware risk monitoring, laying the groundwork for future applications in supply chain risk management.
Funding information in the publication:
We gratefully acknowledge the financial support provided by Business Finland, as this research was carried out within the framework of the AI-SIM project.
