TY - CHAP

T1 - BENGAL

T2 - 11th International Natural Language Generation Conference, INLG 2018

AU - Ngomo, Axel Cyrille Ngoma

AU - Röder, Michael

AU - Moussallem, Diego

AU - Usbeck, Ricardo

AU - Speck, René

N1 - Horizon 2020 Framework Programme Number: 688227 Publisher Copyright: ©2018 Association for Computational Linguistics

PY - 2018/11/1

Y1 - 2018/11/1

N2 - The manual creation of gold standards for named entity recognition and entity linking is time- and resource-intensive. Moreover, recent works show that such gold standards contain a large proportion of mistakes in addition to being difficult to maintain. We hence present BENGAL, a novel automatic generation of such gold standards as a complement to manually created benchmarks. The main advantage of our benchmarks is that they can be readily generated at any time. They are also cost-effective while being guaranteed to be free of annotation errors. We compare the performance of 11 tools on benchmarks in English generated by BENGAL and on 16 benchmarks created manually. We show that our approach can be ported easily across languages by presenting results achieved by 4 tools on both Brazilian Portuguese and Spanish. Overall, our results suggest that our automatic benchmark generation approach can create varied benchmarks that have characteristics similar to those of existing benchmarks. Our approach is open-source. Our experimental results are available at http://faturl.com/bengalexpinlg and the code at https://github.com/dice-group/BENGAL.

AB - The manual creation of gold standards for named entity recognition and entity linking is time- and resource-intensive. Moreover, recent works show that such gold standards contain a large proportion of mistakes in addition to being difficult to maintain. We hence present BENGAL, a novel automatic generation of such gold standards as a complement to manually created benchmarks. The main advantage of our benchmarks is that they can be readily generated at any time. They are also cost-effective while being guaranteed to be free of annotation errors. We compare the performance of 11 tools on benchmarks in English generated by BENGAL and on 16 benchmarks created manually. We show that our approach can be ported easily across languages by presenting results achieved by 4 tools on both Brazilian Portuguese and Spanish. Overall, our results suggest that our automatic benchmark generation approach can create varied benchmarks that have characteristics similar to those of existing benchmarks. Our approach is open-source. Our experimental results are available at http://faturl.com/bengalexpinlg and the code at https://github.com/dice-group/BENGAL.

KW - Informatics

KW - Business informatics

UR - http://www.scopus.com/inward/record.url?scp=85066903644&partnerID=8YFLogxK

UR - https://www.mendeley.com/catalogue/88bd5c19-8006-3731-90cb-b1c32ea66b96/

U2 - 10.18653/v1/W18-6541

DO - 10.18653/v1/W18-6541

M3 - Article in conference proceedings

AN - SCOPUS:85066903644

T3 - INLG 2018 - 11th International Natural Language Generation Conference, Proceedings of the Conference

SP - 339

EP - 349

BT - INLG 2018 - 11th International Natural Language Generation Conference, Proceedings of the Conference

A2 - Krahmer, Emiel

A2 - Gatt, Albert

A2 - Goudbeek, Martijn

PB - Association for Computational Linguistics (ACL)

Y2 - 5 November 2018 through 8 November 2018

ER -