TY - CHAP

T1 - QUANT - Question Answering Benchmark Curator

AU - Gusmita, Ria Hari

AU - Jalota, Rricha

AU - Vollmers, Daniel

AU - Reineke, Jan

AU - Ngonga Ngomo, Axel Cyrille

AU - Usbeck, Ricardo

N1 - Conference code: 15

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Question answering engines have become one of the most popular type of applications driven by Semantic Web technologies. Consequently, the provision of means to quantify the performance of current question answering approaches on current datasets has become ever more important. However, a large percentage of the queries found in popular question answering benchmarks cannot be executed on current versions of their reference dataset. There is a consequently a clear need to curate question answering benchmarks periodically. However, the manual alteration of question answering benchmarks is often error-prone. We alleviate this problem by presenting QUANT, a novel framework for the creation and curation of question answering benchmarks. QUANT supports the curation of benchmarks by generating smart edit suggestions for question-query pair and for the corresponding metadata. In addition, our framework supports the creation of new benchmark entries by providing predefined quality checks for queries. We evaluate QUANT on 653 questions obtained from QALD-1 to QALD-8 with 10 users. Our results show that our framework generates reliable suggestions and can reduce the curation effort for QA benchmarks by up to 91%.

AB - Question answering engines have become one of the most popular type of applications driven by Semantic Web technologies. Consequently, the provision of means to quantify the performance of current question answering approaches on current datasets has become ever more important. However, a large percentage of the queries found in popular question answering benchmarks cannot be executed on current versions of their reference dataset. There is a consequently a clear need to curate question answering benchmarks periodically. However, the manual alteration of question answering benchmarks is often error-prone. We alleviate this problem by presenting QUANT, a novel framework for the creation and curation of question answering benchmarks. QUANT supports the curation of benchmarks by generating smart edit suggestions for question-query pair and for the corresponding metadata. In addition, our framework supports the creation of new benchmark entries by providing predefined quality checks for queries. We evaluate QUANT on 653 questions obtained from QALD-1 to QALD-8 with 10 users. Our results show that our framework generates reliable suggestions and can reduce the curation effort for QA benchmarks by up to 91%.

KW - Benchmark

KW - Knowledge base

KW - Question answering

KW - Informatics

KW - Business informatics

UR - http://www.scopus.com/inward/record.url?scp=85076221758&partnerID=8YFLogxK

UR - https://www.mendeley.com/catalogue/bd2d8832-d7a4-3b1e-99bc-89ac85f1e4bf/

U2 - 10.1007/978-3-030-33220-4_25

DO - 10.1007/978-3-030-33220-4_25

M3 - Article in conference proceedings

AN - SCOPUS:85076221758

SN - 978-3-030-33219-8

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 343

EP - 358

BT - Semantic Systems. The Power of AI and Knowledge Graphs - 15th International Conference, SEMANTiCS 2019 Proceedings

A2 - Acosta, Maribel

A2 - Sure-Vetter, York

A2 - Cudré-Mauroux, Philippe

A2 - Maleshkova, Maria

A2 - Pellegrini, Tassilo

A2 - Sack, Harald

PB - Springer

T2 - 15th International Conference on Semantic Systems, SEMANTiCS 2019

Y2 - 9 September 2019 through 12 September 2019

ER -