A4 Vertaisarvioitu artikkeli konferenssijulkaisussa

Towards better structured and less noisy Web data: Oscar with Register annotations




TekijätLaippala Veronika, Salmela Anna, Rönnqvist Samuel, Aji Alham Fikri, Chang Li-Hsin, Dhifallah Asma, Goulart Larissa, Kortelainen Henna, Pàmies Marc, Prina Dutra Deise, Skantsi Valtteri, Sutawika Lingtang, Pyysalo Sampo

ToimittajaN/A

Konferenssin vakiintunut nimiInternational Conference on Computational Linguistics

Julkaisuvuosi2022

JournalInternational Conference on Computational Linguistics

Kokoomateoksen nimiProceedings of the Eighth Workshop on Noisy User-generated Text (W-NUT 2022)

Sarjan nimiInternational Conference on Computational Linguistics

Numero sarjassa4

Vuosikerta29

Aloitussivu215

Lopetussivu221

ISSN 2951-2093

Verkko-osoitehttps://aclanthology.org/2022.wnut-1.23/

Rinnakkaistallenteen osoitehttps://research.utu.fi/converis/portal/detail/Publication/177823149


Tiivistelmä

Web-crawled datasets are known to be noisy, as they feature a wide range of language use covering both user-generated and professionally edited content as well as noise originating from the crawling process. This article presents one solution to reduce this noise by using automatic register (genre) identification -whether the texts are, e.g., forum discussions, lyrical or how-to pages. We apply the multilingual register identification model by Rönnqvist et al. (2021) and label the widely used Oscar dataset. Additionally, we evaluate the model against eight new languages, showing that the performance is comparable to previous findings on a restricted set of languages. Finally, we present and apply a machine learning method for further cleaning text files originating from Web crawls from remains of boilerplate and other elements not belonging to the main text of the Web page. The register labeled and cleaned dataset covers 351 million documents in 14 languages and is available at https://huggingface.co/datasets/TurkuNLP/register_oscar.


Ladattava julkaisu

This is an electronic reprint of the original article.
This reprint may differ from the original in pagination and typographic detail. Please cite the original version.





Last updated on 2024-26-11 at 13:14