BENGAL: An automatic benchmark generator for entity recognition and linking
Research output: Contributions to collected editions/works › Article in conference proceedings › Research › peer-review
Standard
INLG 2018 - 11th International Natural Language Generation Conference, Proceedings of the Conference. ed. / Emiel Krahmer; Albert Gatt; Martijn Goudbeek. Association for Computational Linguistics (ACL), 2018. p. 339-349 (INLG 2018 - 11th International Natural Language Generation Conference, Proceedings of the Conference).
Research output: Contributions to collected editions/works › Article in conference proceedings › Research › peer-review
Harvard
APA
Vancouver
Bibtex
}
RIS
TY - CHAP
T1 - BENGAL
T2 - 11th International Natural Language Generation Conference, INLG 2018
AU - Ngomo, Axel Cyrille Ngoma
AU - Röder, Michael
AU - Moussallem, Diego
AU - Usbeck, Ricardo
AU - Speck, René
N1 - Horizon 2020 Framework Programme Number: 688227 Publisher Copyright: ©2018 Association for Computational Linguistics
PY - 2018/11/1
Y1 - 2018/11/1
N2 - The manual creation of gold standards for named entity recognition and entity linking is time- and resource-intensive. Moreover, recent works show that such gold standards contain a large proportion of mistakes in addition to being difficult to maintain. We hence present BENGAL, a novel automatic generation of such gold standards as a complement to manually created benchmarks. The main advantage of our benchmarks is that they can be readily generated at any time. They are also cost-effective while being guaranteed to be free of annotation errors. We compare the performance of 11 tools on benchmarks in English generated by BENGAL and on 16 benchmarks created manually. We show that our approach can be ported easily across languages by presenting results achieved by 4 tools on both Brazilian Portuguese and Spanish. Overall, our results suggest that our automatic benchmark generation approach can create varied benchmarks that have characteristics similar to those of existing benchmarks. Our approach is open-source. Our experimental results are available at http://faturl.com/bengalexpinlg and the code at https://github.com/dice-group/BENGAL.
AB - The manual creation of gold standards for named entity recognition and entity linking is time- and resource-intensive. Moreover, recent works show that such gold standards contain a large proportion of mistakes in addition to being difficult to maintain. We hence present BENGAL, a novel automatic generation of such gold standards as a complement to manually created benchmarks. The main advantage of our benchmarks is that they can be readily generated at any time. They are also cost-effective while being guaranteed to be free of annotation errors. We compare the performance of 11 tools on benchmarks in English generated by BENGAL and on 16 benchmarks created manually. We show that our approach can be ported easily across languages by presenting results achieved by 4 tools on both Brazilian Portuguese and Spanish. Overall, our results suggest that our automatic benchmark generation approach can create varied benchmarks that have characteristics similar to those of existing benchmarks. Our approach is open-source. Our experimental results are available at http://faturl.com/bengalexpinlg and the code at https://github.com/dice-group/BENGAL.
KW - Informatics
KW - Business informatics
UR - http://www.scopus.com/inward/record.url?scp=85066903644&partnerID=8YFLogxK
UR - https://www.mendeley.com/catalogue/88bd5c19-8006-3731-90cb-b1c32ea66b96/
U2 - 10.18653/v1/W18-6541
DO - 10.18653/v1/W18-6541
M3 - Article in conference proceedings
AN - SCOPUS:85066903644
T3 - INLG 2018 - 11th International Natural Language Generation Conference, Proceedings of the Conference
SP - 339
EP - 349
BT - INLG 2018 - 11th International Natural Language Generation Conference, Proceedings of the Conference
A2 - Krahmer, Emiel
A2 - Gatt, Albert
A2 - Goudbeek, Martijn
PB - Association for Computational Linguistics (ACL)
Y2 - 5 November 2018 through 8 November 2018
ER -