A4 Refereed article in a conference publication
Automating Qualitative Data Analysis with Chain-of-Thought Reasoning Models: A Study with the Gioia Method
Authors: Laato, Joonatan; Mäntymäki, Matti; Kordyaka, Bastian; Laato, Samuli
Editors: N/A
Conference name: Americas Conference on Information Systems
Publisher: Association for Information Systems
Publication year: 2025
Journal: Americas Conference on Information Systems
Book title : 2025 Americas Conference on Information Systems, AMCIS 2025
First page : 1942
Last page: 1951
ISBN: 978-83-313-2774-4
ISSN: 3066-8743
eISSN: 3066-876X
Publication's open availability at the time of reporting: No Open Access
Publication channel's open availability : No Open Access publication channel
Web address : https://aisel.aisnet.org/amcis2025/data_science/sig_dsa/1
In this study, we explore automating the analysis steps in the procedurally rigorous qualitative analysis approach popular in the field of IS: the Gioia method. Using DeepSeek's R1 chain-of-thought model, custom-built pipelines on a high-performance computing infrastructure, we sought to replicate a peer-reviewed and published original analysis of 17 expert interview transcripts. To address model hallucinations, we used Levenshtein distance to check that provided quotes exist in the transcript. We found that while the constructed pipeline produced concepts similar to the ones in the original publication, there were challenges in maintaining interpretive rigor (i.e., the system extracted ist order concepts but lost meanings associated with them). This led the LLM to overgeneralize, ending up with 2nd order themes and aggregate dimensions that were inaccurate and borderline non-informative. We elaborate on nine unique challenges we encountered and provide directions for future research.
