A4 Vertaisarvioitu artikkeli konferenssijulkaisussa
Automating Qualitative Data Analysis with Chain-of-Thought Reasoning Models: A Study with the Gioia Method
Tekijät: Laato, Joonatan; Mäntymäki, Matti; Kordyaka, Bastian; Laato, Samuli
Toimittaja: N/A
Konferenssin vakiintunut nimi: Americas Conference on Information Systems
Kustantaja: Association for Information Systems
Julkaisuvuosi: 2025
Lehti: Americas Conference on Information Systems
Kokoomateoksen nimi: 2025 Americas Conference on Information Systems, AMCIS 2025
Aloitussivu: 1942
Lopetussivu: 1951
ISBN: 978-83-313-2774-4
ISSN: 3066-8743
eISSN: 3066-876X
Julkaisun avoimuus kirjaamishetkellä: Ei avoimesti saatavilla
Julkaisukanavan avoimuus : Ei avoin julkaisukanava
Verkko-osoite: https://aisel.aisnet.org/amcis2025/data_science/sig_dsa/1
In this study, we explore automating the analysis steps in the procedurally rigorous qualitative analysis approach popular in the field of IS: the Gioia method. Using DeepSeek's R1 chain-of-thought model, custom-built pipelines on a high-performance computing infrastructure, we sought to replicate a peer-reviewed and published original analysis of 17 expert interview transcripts. To address model hallucinations, we used Levenshtein distance to check that provided quotes exist in the transcript. We found that while the constructed pipeline produced concepts similar to the ones in the original publication, there were challenges in maintaining interpretive rigor (i.e., the system extracted ist order concepts but lost meanings associated with them). This led the LLM to overgeneralize, ending up with 2nd order themes and aggregate dimensions that were inaccurate and borderline non-informative. We elaborate on nine unique challenges we encountered and provide directions for future research.
