QUANT - Question Answering Benchmark Curator

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

  • Ria Hari Gusmita
  • Rricha Jalota
  • Daniel Vollmers
  • Jan Reineke
  • Axel Cyrille Ngonga Ngomo
  • Ricardo Usbeck

Question answering engines have become one of the most popular type of applications driven by Semantic Web technologies. Consequently, the provision of means to quantify the performance of current question answering approaches on current datasets has become ever more important. However, a large percentage of the queries found in popular question answering benchmarks cannot be executed on current versions of their reference dataset. There is a consequently a clear need to curate question answering benchmarks periodically. However, the manual alteration of question answering benchmarks is often error-prone. We alleviate this problem by presenting QUANT, a novel framework for the creation and curation of question answering benchmarks. QUANT supports the curation of benchmarks by generating smart edit suggestions for question-query pair and for the corresponding metadata. In addition, our framework supports the creation of new benchmark entries by providing predefined quality checks for queries. We evaluate QUANT on 653 questions obtained from QALD-1 to QALD-8 with 10 users. Our results show that our framework generates reliable suggestions and can reduce the curation effort for QA benchmarks by up to 91%.

Original languageEnglish
Title of host publicationSemantic Systems. The Power of AI and Knowledge Graphs - 15th International Conference, SEMANTiCS 2019 Proceedings
EditorsMaribel Acosta, York Sure-Vetter, Philippe Cudré-Mauroux, Maria Maleshkova, Tassilo Pellegrini, Harald Sack
Number of pages16
PublisherSpringer
Publication date01.01.2019
Pages343-358
ISBN (print)978-3-030-33219-8
ISBN (electronic)978-3-030-33220-4
DOIs
Publication statusPublished - 01.01.2019
Externally publishedYes
Event15th International Conference on Semantic Systems, SEMANTiCS 2019 - Karlsruhe, Germany
Duration: 09.09.201912.09.2019
Conference number: 15
https://2019.semantics.cc/

Bibliographical note

This work was supported by the German Federal Ministry of Transport and Digital Infrastructure (BMVI) in the project LIMBO (no. 19F2029I) and by the German Federal Ministry of Education and Research (BMBF) in the project SOLIDE (no. 13N14456) within ‘KMU-innovativ: Forschung für die zivile Sicherheit’ in particular ‘Forschung für die zivile Sicherheit’.

Publisher Copyright:
© 2019, The Author(s).