BENGAL: An automatic benchmark generator for entity recognition and linking

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

  • Axel Cyrille Ngoma Ngomo
  • Michael Röder
  • Diego Moussallem
  • Ricardo Usbeck
  • René Speck

The manual creation of gold standards for named entity recognition and entity linking is time- and resource-intensive. Moreover, recent works show that such gold standards contain a large proportion of mistakes in addition to being difficult to maintain. We hence present BENGAL, a novel automatic generation of such gold standards as a complement to manually created benchmarks. The main advantage of our benchmarks is that they can be readily generated at any time. They are also cost-effective while being guaranteed to be free of annotation errors. We compare the performance of 11 tools on benchmarks in English generated by BENGAL and on 16 benchmarks created manually. We show that our approach can be ported easily across languages by presenting results achieved by 4 tools on both Brazilian Portuguese and Spanish. Overall, our results suggest that our automatic benchmark generation approach can create varied benchmarks that have characteristics similar to those of existing benchmarks. Our approach is open-source. Our experimental results are available at http://faturl.com/bengalexpinlg and the code at https://github.com/dice-group/BENGAL.

Original languageEnglish
Title of host publicationINLG 2018 - 11th International Natural Language Generation Conference, Proceedings of the Conference
EditorsEmiel Krahmer, Albert Gatt, Martijn Goudbeek
Number of pages11
PublisherAssociation for Computational Linguistics (ACL)
Publication date01.11.2018
Pages339-349
ISBN (electronic)9781948087865
DOIs
Publication statusPublished - 01.11.2018
Externally publishedYes
Event11th International Natural Language Generation Conference, INLG 2018 - Tilburg Universität , Tilburg, Netherlands
Duration: 05.11.201808.11.2018
https://inlg2018.uvt.nl/#:~:text=The%2011th%20International%20Conference%20on,organised%20in%20nearby%20Brussels%2C%20Belgium.

Bibliographical note

Horizon 2020 Framework Programme Number: 688227

Publisher Copyright:
©2018 Association for Computational Linguistics

Links

DOI