A4 Vertaisarvioitu artikkeli konferenssijulkaisussa

Extracting Social Connections from Finnish Karelian Refugee Interviews Using LLMs




TekijätLaato, Joonatan; Kanerva, Jenna; Loehr, John; Lummaa, Virpi; Ginter, Filip

ToimittajaHaverals, Wouter; Koolen, Marijn; Thompson, Laure

Konferenssin vakiintunut nimiComputational Humanities Research

Julkaisuvuosi2024

JournalCEUR Workshop Proceedings

Kokoomateoksen nimiProceedings of the Computational Humanities Research Conference 2024 (CHR 2024), Aarhus, Denmark, December 4-6, 202

eISSN1613-0073

Verkko-osoitehttps://ceur-ws.org/Vol-3834/paper52.pdf

Rinnakkaistallenteen osoitehttps://research.utu.fi/converis/portal/detail/Publication/470957606


Tiivistelmä

We performed a zero-shot information extraction study on a historical collection of 89,339 brief Finnish-language interviews of refugee families relocated post-WWII from Finnish Eastern Karelia. Our research objective is two-fold. First, we aim to extract social organizations and hobbies from the free text of the interviews, separately for each family member. These can act as a proxy variable indicating the degree of social integration of refugees in their new environment. Second, we aim to evaluate several alternative ways to approach this task, comparing a number of generative models and a supervised learning approach, to gain a broader insight into the relative merits of these different approaches and their applicability in similar studies. We find that the best generative model (GPT-4) is roughly on par with human performance, at an F-score of 88.8%. Interestingly, the best open generative model (Llama-3-70B-Instruct) reaches almost the same performance, at 87.7% F-score, demonstrating that open models are becoming a viable alternative for some practical tasks even on non-English data. Additionally, we test a supervised learning alternative, where we fine-tune a Finnish BERT model (FinBERT) using GPT-4 generated training data. By this method, we achieved an F-score of 84.1% already with 6K interviews up to an F-score of 86.3% with 30k interviews. Such an approach would be particularly appealing in cases where the computational resources are limited, or there is a substantial mass of data to process.


Ladattava julkaisu

This is an electronic reprint of the original article.
This reprint may differ from the original in pagination and typographic detail. Please cite the original version.




Julkaisussa olevat rahoitustiedot
This work was carried out in the Human Diversity University profilation programme (PROFI-7) of the Research Council of Finland, and in part supported also through the Behind the Words general research grant of the Research Council of Finland and the KinSocieties, ERC-2022-ADG grant number 101098266.


Last updated on 2025-27-01 at 19:47