BENGAL: An automatic benchmark generator for entity recognition and linking

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Standard

BENGAL: An automatic benchmark generator for entity recognition and linking. / Ngomo, Axel Cyrille Ngoma; Röder, Michael; Moussallem, Diego et al.
INLG 2018 - 11th International Natural Language Generation Conference, Proceedings of the Conference. ed. / Emiel Krahmer; Albert Gatt; Martijn Goudbeek. Association for Computational Linguistics (ACL), 2018. p. 339-349 (INLG 2018 - 11th International Natural Language Generation Conference, Proceedings of the Conference).

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Harvard

Ngomo, ACN, Röder, M, Moussallem, D, Usbeck, R & Speck, R 2018, BENGAL: An automatic benchmark generator for entity recognition and linking. in E Krahmer, A Gatt & M Goudbeek (eds), INLG 2018 - 11th International Natural Language Generation Conference, Proceedings of the Conference. INLG 2018 - 11th International Natural Language Generation Conference, Proceedings of the Conference, Association for Computational Linguistics (ACL), pp. 339-349, 11th International Natural Language Generation Conference, INLG 2018, Tilburg, Netherlands, 05.11.18. https://doi.org/10.18653/v1/W18-6541

APA

Ngomo, A. C. N., Röder, M., Moussallem, D., Usbeck, R., & Speck, R. (2018). BENGAL: An automatic benchmark generator for entity recognition and linking. In E. Krahmer, A. Gatt, & M. Goudbeek (Eds.), INLG 2018 - 11th International Natural Language Generation Conference, Proceedings of the Conference (pp. 339-349). (INLG 2018 - 11th International Natural Language Generation Conference, Proceedings of the Conference). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/W18-6541

Vancouver

Ngomo ACN, Röder M, Moussallem D, Usbeck R, Speck R. BENGAL: An automatic benchmark generator for entity recognition and linking. In Krahmer E, Gatt A, Goudbeek M, editors, INLG 2018 - 11th International Natural Language Generation Conference, Proceedings of the Conference. Association for Computational Linguistics (ACL). 2018. p. 339-349. (INLG 2018 - 11th International Natural Language Generation Conference, Proceedings of the Conference). doi: 10.18653/v1/W18-6541

Bibtex

@inbook{db4280b15b2e46089944c70f84e0ecf8,
title = "BENGAL: An automatic benchmark generator for entity recognition and linking",
abstract = "The manual creation of gold standards for named entity recognition and entity linking is time- and resource-intensive. Moreover, recent works show that such gold standards contain a large proportion of mistakes in addition to being difficult to maintain. We hence present BENGAL, a novel automatic generation of such gold standards as a complement to manually created benchmarks. The main advantage of our benchmarks is that they can be readily generated at any time. They are also cost-effective while being guaranteed to be free of annotation errors. We compare the performance of 11 tools on benchmarks in English generated by BENGAL and on 16 benchmarks created manually. We show that our approach can be ported easily across languages by presenting results achieved by 4 tools on both Brazilian Portuguese and Spanish. Overall, our results suggest that our automatic benchmark generation approach can create varied benchmarks that have characteristics similar to those of existing benchmarks. Our approach is open-source. Our experimental results are available at http://faturl.com/bengalexpinlg and the code at https://github.com/dice-group/BENGAL.",
keywords = "Informatics, Business informatics",
author = "Ngomo, {Axel Cyrille Ngoma} and Michael R{\"o}der and Diego Moussallem and Ricardo Usbeck and Ren{\'e} Speck",
note = "Horizon 2020 Framework Programme Number: 688227 Publisher Copyright: {\textcopyright}2018 Association for Computational Linguistics; 11th International Natural Language Generation Conference, INLG 2018 ; Conference date: 05-11-2018 Through 08-11-2018",
year = "2018",
month = nov,
day = "1",
doi = "10.18653/v1/W18-6541",
language = "English",
series = "INLG 2018 - 11th International Natural Language Generation Conference, Proceedings of the Conference",
publisher = "Association for Computational Linguistics (ACL)",
pages = "339--349",
editor = "Emiel Krahmer and Albert Gatt and Martijn Goudbeek",
booktitle = "INLG 2018 - 11th International Natural Language Generation Conference, Proceedings of the Conference",
address = "United States",
url = "https://inlg2018.uvt.nl/#:~:text=The%2011th%20International%20Conference%20on,organised%20in%20nearby%20Brussels%2C%20Belgium.",

}

RIS

TY - CHAP

T1 - BENGAL

T2 - 11th International Natural Language Generation Conference, INLG 2018

AU - Ngomo, Axel Cyrille Ngoma

AU - Röder, Michael

AU - Moussallem, Diego

AU - Usbeck, Ricardo

AU - Speck, René

N1 - Horizon 2020 Framework Programme Number: 688227 Publisher Copyright: ©2018 Association for Computational Linguistics

PY - 2018/11/1

Y1 - 2018/11/1

N2 - The manual creation of gold standards for named entity recognition and entity linking is time- and resource-intensive. Moreover, recent works show that such gold standards contain a large proportion of mistakes in addition to being difficult to maintain. We hence present BENGAL, a novel automatic generation of such gold standards as a complement to manually created benchmarks. The main advantage of our benchmarks is that they can be readily generated at any time. They are also cost-effective while being guaranteed to be free of annotation errors. We compare the performance of 11 tools on benchmarks in English generated by BENGAL and on 16 benchmarks created manually. We show that our approach can be ported easily across languages by presenting results achieved by 4 tools on both Brazilian Portuguese and Spanish. Overall, our results suggest that our automatic benchmark generation approach can create varied benchmarks that have characteristics similar to those of existing benchmarks. Our approach is open-source. Our experimental results are available at http://faturl.com/bengalexpinlg and the code at https://github.com/dice-group/BENGAL.

AB - The manual creation of gold standards for named entity recognition and entity linking is time- and resource-intensive. Moreover, recent works show that such gold standards contain a large proportion of mistakes in addition to being difficult to maintain. We hence present BENGAL, a novel automatic generation of such gold standards as a complement to manually created benchmarks. The main advantage of our benchmarks is that they can be readily generated at any time. They are also cost-effective while being guaranteed to be free of annotation errors. We compare the performance of 11 tools on benchmarks in English generated by BENGAL and on 16 benchmarks created manually. We show that our approach can be ported easily across languages by presenting results achieved by 4 tools on both Brazilian Portuguese and Spanish. Overall, our results suggest that our automatic benchmark generation approach can create varied benchmarks that have characteristics similar to those of existing benchmarks. Our approach is open-source. Our experimental results are available at http://faturl.com/bengalexpinlg and the code at https://github.com/dice-group/BENGAL.

KW - Informatics

KW - Business informatics

UR - http://www.scopus.com/inward/record.url?scp=85066903644&partnerID=8YFLogxK

UR - https://www.mendeley.com/catalogue/88bd5c19-8006-3731-90cb-b1c32ea66b96/

U2 - 10.18653/v1/W18-6541

DO - 10.18653/v1/W18-6541

M3 - Article in conference proceedings

AN - SCOPUS:85066903644

T3 - INLG 2018 - 11th International Natural Language Generation Conference, Proceedings of the Conference

SP - 339

EP - 349

BT - INLG 2018 - 11th International Natural Language Generation Conference, Proceedings of the Conference

A2 - Krahmer, Emiel

A2 - Gatt, Albert

A2 - Goudbeek, Martijn

PB - Association for Computational Linguistics (ACL)

Y2 - 5 November 2018 through 8 November 2018

ER -

Links

DOI