A4 Refereed article in a conference publication
On variability in the identification and labelling of disfluencies — preliminary results from 23 annotations of the same data
Authors: Trouvain, Jürgen; Crible, Ludivine; Belz, Malte; Betz, Simon; Beňuš, Štefan; Baqué, Lorraine; Cantarutti, Marina; Di Napoli, Jessica; Didirková, Ivana; Machuca, Maria; Mareková, Lucia; Niculescu, Oana; Peltonen, Pauliina; Pistono, Aurelie; Schettino, Loredana; Silber-Varod, Vered; Williams, Simon
Editors: Helena Moniz and Fernando Batista
Conference name: Disfluency in Spontaneous Speech Workshop
Publication year: 2025
Journal: Interspeech
Book title : Proc. Disfluency in Spontaneous Speech (DiSS) Workshop 2025
First page : 57
Last page: 61
DOI: https://doi.org/10.21437/DiSS.2025-12
Publication's open availability at the time of reporting: Open Access
Publication channel's open availability : Open Access publication channel
Web address : https://www.isca-archive.org/tmp/diss_2025/trouvain25_diss.html
Self-archived copy’s web address: https://research.utu.fi/converis/portal/detail/Publication/499212588
Self-archived copy's version: Publisher`s PDF
This study provides a preliminary report on a large inter-annotator agreement experiment where 23 expert annotators from various research backgrounds identified and labelled disfluencies in the same speech sample. Each annotator was instructed to analyze the sample according to the framework (definitions, segmentation, labels, etc.) they typically use. The annotations were then processed and compared across three different dimensions: 1) the scope of the chosen typology and the definitions within, 2) the implementation of the typology in terms of annotation tiers and labels, and 3) the temporal alignment of the annotations. Preliminary findings reveal that there are substantial variations between annotators on various levels of annotation. The lack of a common standard becomes particularly evident in more complex segments, such as repairs.
Downloadable publication This is an electronic reprint of the original article. |  |
