Standard
QUANT - Question Answering Benchmark Curator. / Gusmita, Ria Hari; Jalota, Rricha; Vollmers, Daniel et al.
Semantic Systems. The Power of AI and Knowledge Graphs - 15th International Conference, SEMANTiCS 2019 Proceedings. ed. / Maribel Acosta; York Sure-Vetter; Philippe Cudré-Mauroux; Maria Maleshkova; Tassilo Pellegrini; Harald Sack. Springer, 2019. p. 343-358 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11702 LNCS).
Research output: Contributions to collected editions/works › Article in conference proceedings › Research › peer-review
Harvard
Gusmita, RH, Jalota, R, Vollmers, D, Reineke, J, Ngonga Ngomo, AC
& Usbeck, R 2019,
QUANT - Question Answering Benchmark Curator. in M Acosta, Y Sure-Vetter, P Cudré-Mauroux, M Maleshkova, T Pellegrini & H Sack (eds),
Semantic Systems. The Power of AI and Knowledge Graphs - 15th International Conference, SEMANTiCS 2019 Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11702 LNCS, Springer, pp. 343-358, 15th International Conference on Semantic Systems, SEMANTiCS 2019, Karlsruhe, Germany,
09.09.19.
https://doi.org/10.1007/978-3-030-33220-4_25APA
Gusmita, R. H., Jalota, R., Vollmers, D., Reineke, J., Ngonga Ngomo, A. C.
, & Usbeck, R. (2019).
QUANT - Question Answering Benchmark Curator. In M. Acosta, Y. Sure-Vetter, P. Cudré-Mauroux, M. Maleshkova, T. Pellegrini, & H. Sack (Eds.),
Semantic Systems. The Power of AI and Knowledge Graphs - 15th International Conference, SEMANTiCS 2019 Proceedings (pp. 343-358). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11702 LNCS). Springer.
https://doi.org/10.1007/978-3-030-33220-4_25Vancouver
Gusmita RH, Jalota R, Vollmers D, Reineke J, Ngonga Ngomo AC
, Usbeck R.
QUANT - Question Answering Benchmark Curator. In Acosta M, Sure-Vetter Y, Cudré-Mauroux P, Maleshkova M, Pellegrini T, Sack H, editors, Semantic Systems. The Power of AI and Knowledge Graphs - 15th International Conference, SEMANTiCS 2019 Proceedings. Springer. 2019. p. 343-358. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-33220-4_25
Bibtex
@inbook{31b33b4964fc4fc4a7582f0970a1217d,
title = "QUANT - Question Answering Benchmark Curator",
abstract = "Question answering engines have become one of the most popular type of applications driven by Semantic Web technologies. Consequently, the provision of means to quantify the performance of current question answering approaches on current datasets has become ever more important. However, a large percentage of the queries found in popular question answering benchmarks cannot be executed on current versions of their reference dataset. There is a consequently a clear need to curate question answering benchmarks periodically. However, the manual alteration of question answering benchmarks is often error-prone. We alleviate this problem by presenting QUANT, a novel framework for the creation and curation of question answering benchmarks. QUANT supports the curation of benchmarks by generating smart edit suggestions for question-query pair and for the corresponding metadata. In addition, our framework supports the creation of new benchmark entries by providing predefined quality checks for queries. We evaluate QUANT on 653 questions obtained from QALD-1 to QALD-8 with 10 users. Our results show that our framework generates reliable suggestions and can reduce the curation effort for QA benchmarks by up to 91%.",
keywords = "Benchmark, Knowledge base, Question answering, Informatics, Business informatics",
author = "Gusmita, {Ria Hari} and Rricha Jalota and Daniel Vollmers and Jan Reineke and {Ngonga Ngomo}, {Axel Cyrille} and Ricardo Usbeck",
note = "This work was supported by the German Federal Ministry of Transport and Digital Infrastructure (BMVI) in the project LIMBO (no. 19F2029I) and by the German Federal Ministry of Education and Research (BMBF) in the project SOLIDE (no. 13N14456) within {\textquoteleft}KMU-innovativ: Forschung f{\"u}r die zivile Sicherheit{\textquoteright} in particular {\textquoteleft}Forschung f{\"u}r die zivile Sicherheit{\textquoteright}. Publisher Copyright: {\textcopyright} 2019, The Author(s).; 15th International Conference on Semantic Systems, SEMANTiCS 2019, SEMANTiCS 2019 ; Conference date: 09-09-2019 Through 12-09-2019",
year = "2019",
month = jan,
day = "1",
doi = "10.1007/978-3-030-33220-4_25",
language = "English",
isbn = "978-3-030-33219-8",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer",
pages = "343--358",
editor = "Maribel Acosta and York Sure-Vetter and Philippe Cudr{\'e}-Mauroux and Maria Maleshkova and Tassilo Pellegrini and Harald Sack",
booktitle = "Semantic Systems. The Power of AI and Knowledge Graphs - 15th International Conference, SEMANTiCS 2019 Proceedings",
address = "Germany",
url = "https://2019.semantics.cc/",
}
RIS
TY - CHAP
T1 - QUANT - Question Answering Benchmark Curator
AU - Gusmita, Ria Hari
AU - Jalota, Rricha
AU - Vollmers, Daniel
AU - Reineke, Jan
AU - Ngonga Ngomo, Axel Cyrille
AU - Usbeck, Ricardo
N1 - Conference code: 15
PY - 2019/1/1
Y1 - 2019/1/1
N2 - Question answering engines have become one of the most popular type of applications driven by Semantic Web technologies. Consequently, the provision of means to quantify the performance of current question answering approaches on current datasets has become ever more important. However, a large percentage of the queries found in popular question answering benchmarks cannot be executed on current versions of their reference dataset. There is a consequently a clear need to curate question answering benchmarks periodically. However, the manual alteration of question answering benchmarks is often error-prone. We alleviate this problem by presenting QUANT, a novel framework for the creation and curation of question answering benchmarks. QUANT supports the curation of benchmarks by generating smart edit suggestions for question-query pair and for the corresponding metadata. In addition, our framework supports the creation of new benchmark entries by providing predefined quality checks for queries. We evaluate QUANT on 653 questions obtained from QALD-1 to QALD-8 with 10 users. Our results show that our framework generates reliable suggestions and can reduce the curation effort for QA benchmarks by up to 91%.
AB - Question answering engines have become one of the most popular type of applications driven by Semantic Web technologies. Consequently, the provision of means to quantify the performance of current question answering approaches on current datasets has become ever more important. However, a large percentage of the queries found in popular question answering benchmarks cannot be executed on current versions of their reference dataset. There is a consequently a clear need to curate question answering benchmarks periodically. However, the manual alteration of question answering benchmarks is often error-prone. We alleviate this problem by presenting QUANT, a novel framework for the creation and curation of question answering benchmarks. QUANT supports the curation of benchmarks by generating smart edit suggestions for question-query pair and for the corresponding metadata. In addition, our framework supports the creation of new benchmark entries by providing predefined quality checks for queries. We evaluate QUANT on 653 questions obtained from QALD-1 to QALD-8 with 10 users. Our results show that our framework generates reliable suggestions and can reduce the curation effort for QA benchmarks by up to 91%.
KW - Benchmark
KW - Knowledge base
KW - Question answering
KW - Informatics
KW - Business informatics
UR - http://www.scopus.com/inward/record.url?scp=85076221758&partnerID=8YFLogxK
UR - https://www.mendeley.com/catalogue/bd2d8832-d7a4-3b1e-99bc-89ac85f1e4bf/
U2 - 10.1007/978-3-030-33220-4_25
DO - 10.1007/978-3-030-33220-4_25
M3 - Article in conference proceedings
AN - SCOPUS:85076221758
SN - 978-3-030-33219-8
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 343
EP - 358
BT - Semantic Systems. The Power of AI and Knowledge Graphs - 15th International Conference, SEMANTiCS 2019 Proceedings
A2 - Acosta, Maribel
A2 - Sure-Vetter, York
A2 - Cudré-Mauroux, Philippe
A2 - Maleshkova, Maria
A2 - Pellegrini, Tassilo
A2 - Sack, Harald
PB - Springer
T2 - 15th International Conference on Semantic Systems, SEMANTiCS 2019
Y2 - 9 September 2019 through 12 September 2019
ER -
