A4 Refereed article in a conference publication

Why Compromise Privacy? Local LLMs Rival Commercial LLMs in Qualitative Analysis




AuthorsAdeseye, Aisvarya; Isoaho, Jouni; Virtanen, Seppo; Mohammad, Tahir

EditorsN/A

Conference nameComputing, Communications and IoT Applications

Publication year2025

Book title 2025 Computing, Communications and IoT Applications (ComComAp)

First page 127

Last page132

ISBN979-8-3315-9144-1

eISBN979-8-3315-9143-4

DOIhttps://doi.org/10.1109/ComComAp68359.2025.11353130

Publication's open availability at the time of reportingNo Open Access

Publication channel's open availability No Open Access publication channel

Web address https://ieeexplore.ieee.org/document/11353130


Abstract

Large Language Models (LLMs) are increasingly being applied in qualitative analysis for tasks such as theme extraction, frequency analysis, and impact evaluation. However, their adoption raises privacy and GDPR compliance concerns when transcripts are processed using commercial LLMs such as ChatGPT or Gemini. Existing studies highlight these risks but provide little systematic evidence for comparing local and commercial LLMs. This study evaluates the performance of local LLMs such as LLaMA-3.1 (8B), LLaMA-3.2 (1B−3B), LLaMA-3.3 (70B), Gemma-2 (2B−27B), and Phi-3.5 (3.5B−6.6B) against commercial LLMs (ChatGPT-4o and Gemini-2.5 Flash) using 82 anonymized transcripts for qualitative analysis tasks. A structured prompt design was applied, and the results were benchmarked against ground-truth coding using cost, through-put, hallucination rate, and accuracy rate. The findings indicate that the small local LLMs (about 3B) performed comparably close to Gemini, medium models (6-9B) performed close to ChatGPT, and large LLMs (27B−70B) consistently outperformed both commercial LLMs. Hallucination reduction of up to 85% was observed with local LLMs at negligible recurring costs. Furthermore, local LLMs help with GDPR compliance and privacy preservation. It also minimizes cost while delivering accuracy that is comparable, or better than the commonly available commercial LLMs.



Last updated on 30/01/2026 07:20:36 AM