A4 Vertaisarvioitu artikkeli konferenssijulkaisussa
Extracting Social Connections from Finnish Karelian Refugee Interviews Using LLMs
Tekijät: Laato, Joonatan; Kanerva, Jenna; Loehr, John; Lummaa, Virpi; Ginter, Filip
Toimittaja: Haverals, Wouter; Koolen, Marijn; Thompson, Laure
Konferenssin vakiintunut nimi: Computational Humanities Research
Julkaisuvuosi: 2024
Journal: CEUR Workshop Proceedings
Kokoomateoksen nimi: Proceedings of the Computational Humanities Research Conference 2024 (CHR 2024), Aarhus, Denmark, December 4-6, 202
eISSN: 1613-0073
Verkko-osoite: https://ceur-ws.org/Vol-3834/paper52.pdf
Rinnakkaistallenteen osoite: https://research.utu.fi/converis/portal/detail/Publication/470957606
We performed a zero-shot information extraction study on a historical collection of 89,339 brief Finnish-language interviews of refugee families relocated post-WWII from Finnish Eastern Karelia. Our research objective is two-fold. First, we aim to extract social organizations and hobbies from the free text of the interviews, separately for each family member. These can act as a proxy variable indicating the degree of social integration of refugees in their new environment. Second, we aim to evaluate several alternative ways to approach this task, comparing a number of generative models and a supervised learning approach, to gain a broader insight into the relative merits of these different approaches and their applicability in similar studies. We find that the best generative model (GPT-4) is roughly on par with human performance, at an F-score of 88.8%. Interestingly, the best open generative model (Llama-3-70B-Instruct) reaches almost the same performance, at 87.7% F-score, demonstrating that open models are becoming a viable alternative for some practical tasks even on non-English data. Additionally, we test a supervised learning alternative, where we fine-tune a Finnish BERT model (FinBERT) using GPT-4 generated training data. By this method, we achieved an F-score of 84.1% already with 6K interviews up to an F-score of 86.3% with 30k interviews. Such an approach would be particularly appealing in cases where the computational resources are limited, or there is a substantial mass of data to process.
Ladattava julkaisu This is an electronic reprint of the original article. |
Julkaisussa olevat rahoitustiedot:
This work was carried out in the Human Diversity University profilation programme (PROFI-7) of the Research Council of Finland, and in part supported also through the Behind the Words general research grant of the Research Council of Finland and the KinSocieties, ERC-2022-ADG grant number 101098266.