Automating Qualitative Data Analysis with Chain-of-Thought Reasoning Models: A Study with the Gioia Method

: Laato, Joonatan; Mäntymäki, Matti; Kordyaka, Bastian; Laato, Samuli

: N/A

: Americas Conference on Information Systems

Publisher: Association for Information Systems

: 2025

Americas Conference on Information Systems

: 2025 Americas Conference on Information Systems, AMCIS 2025

: 1942

: 1951

: 978-83-313-2774-4

: 3066-8743

: 3066-876X

: https://aisel.aisnet.org/amcis2025/data_science/sig_dsa/1

In this study, we explore automating the analysis steps in the procedurally rigorous qualitative analysis approach popular in the field of IS: the Gioia method. Using DeepSeek's R1 chain-of-thought model, custom-built pipelines on a high-performance computing infrastructure, we sought to replicate a peer-reviewed and published original analysis of 17 expert interview transcripts. To address model hallucinations, we used Levenshtein distance to check that provided quotes exist in the transcript. We found that while the constructed pipeline produced concepts similar to the ones in the original publication, there were challenges in maintaining interpretive rigor (i.e., the system extracted ist order concepts but lost meanings associated with them). This led the LLM to overgeneralize, ending up with 2nd order themes and aggregate dimensions that were inaccurate and borderline non-informative. We elaborate on nine unique challenges we encountered and provide directions for future research.