Refereed article in conference proceedings (A4)

Exploring Cross-sentence Contexts for Named Entity Recognition with BERT

List of Authors: Luoma Jouni, Pyysalo Sampo

Conference name: International Conference on Computational Linguistics

Publication year: 2020

Journal: International Conference on Computational Linguistics

Book title *: Proceedings of the 28th International Conference on Computational Linguistics

ISBN: 978-1-952148-27-9


Self-archived copy’s web address:


Named entity recognition (NER) is frequently addressed as a sequence classification task with
each input consisting of one sentence of text. It is nevertheless clear that useful information for
NER is often found also elsewhere in text. Recent self-attention models like BERT can both
capture long-distance relationships in input and represent inputs consisting of several sentences.
This creates opportunities for adding cross-sentence information in natural language processing
tasks. This paper presents a systematic study exploring the use of cross-sentence information
for NER using BERT models in five languages. We find that adding context as additional sentences to BERT input systematically increases NER performance. Multiple sentences in input
samples allows us to study the predictions of the sentences in different contexts. We propose
a straightforward method, Contextual Majority Voting (CMV), to combine these different predictions and demonstrate this to further increase NER performance. Evaluation on established
datasets, including the CoNLL’02 and CoNLL’03 NER benchmarks, demonstrates that our proposed approach can improve on the state-of-the-art NER results on English, Dutch, and Finnish,
achieves the best reported BERT-based results on German, and is on par with other BERT-based
approaches in Spanish. We release all methods implemented in this work under open licenses.

Downloadable publication

This is an electronic reprint of the original article.
This reprint may differ from the original in pagination and typographic detail. Please cite the original version.

Last updated on 2022-07-04 at 18:19