A Study on the Impact of Intradomain Finetuning of Deep Language Models for Legal Named Entity Recognition in Portuguese

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review


Deep language models, like ELMo, BERT and GPT, have achieved impressive results on several natural language tasks. These models are pretrained on large corpora of unlabeled general domain text and later supervisedly trained on downstream tasks. An optional step consists of finetuning the language model on a large intradomain corpus of unlabeled text, before training it on the final task. This aspect is not well explored in the current literature. In this work, we investigate the impact of this step on named entity recognition (NER) for Portuguese legal documents. We explore different scenarios considering two deep language architectures (ELMo and BERT), four unlabeled corpora and three legal NER tasks for the Portuguese language. Experimental findings show a significant improvement on performance due to language model finetuning on intradomain text. We also evaluate the finetuned models on two general-domain NER tasks, in order to understand whether the aforementioned improvements were really due to domain similarity or simply due to more training data. The achieved results also indicate that finetuning on a legal domain corpus hurts performance on the general-domain NER tasks. Additionally, our BERT model, finetuned on a legal corpus, significantly improves on the state-of-the-art performance on the LeNER-Br corpus, a Portuguese language NER corpus for the legal domain.

Original languageEnglish
Title of host publicationIntelligent Systems : 9th Brazilian Conference, BRACIS 2020, Rio Grande, Brazil, October 20–23, 2020, Proceedings, Part I
EditorsRicardo Cerri, Ronaldo C. Prati
Number of pages15
Place of PublicationCham
PublisherSpringer Nature Switzerland AG
Publication date2020
ISBN (Print)978-3-030-61376-1
ISBN (Electronic)978-3-030-61377-8
Publication statusPublished - 2020
Externally publishedYes
EventBrazilian Conference on Intelligent Systems - BRACIS 2020 - Rio Grande, Brazil
Duration: 20.10.202023.10.2020
Conference number: 9