Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis
Research output: Contributions to collected editions/works › Article in conference proceedings › Research › peer-review
Standard
Knowledge Graph Question Answering Leaderboard : A Community Resource to Prevent a Replication Crisis. / Perevalov, Aleksandr; Yan, Xi; Kovriguina, Liubov et al.
2022 Language Resources and Evaluation Conference, LREC 2022. ed. / Nicoletta Calzolari; Frederic Bechet; Philippe Blache; Khalid Choukri; Christopher Cieri; Thierry Declerck; Sara Goggi; Hitoshi Isahara; Bente Maegaard; Joseph Mariani; Helene Mazo; Jan Odijk; Stelios Piperidis. European Language Resources Association (ELRA), 2022. p. 2998-3007 (2022 Language Resources and Evaluation Conference, LREC 2022).Research output: Contributions to collected editions/works › Article in conference proceedings › Research › peer-review
Harvard
APA
Vancouver
Bibtex
}
RIS
TY - CHAP
T1 - Knowledge Graph Question Answering Leaderboard
T2 - 13th International Conference on Language Resources and Evaluation Conference - LREC 2022
AU - Perevalov, Aleksandr
AU - Yan, Xi
AU - Kovriguina, Liubov
AU - Jiang, Longquan
AU - Both, Andreas
AU - Usbeck, Ricardo
N1 - Conference code: 13
PY - 2022
Y1 - 2022
N2 - Data-driven systems need to be evaluated to establish trust in the scientific approach and its applicability. In particular, this is true for Knowledge Graph (KG) Question Answering (QA), where complex data structures are made accessible via natural-language interfaces. Evaluating the capabilities of these systems has been a driver for the community for more than ten years while establishing different KGQA benchmark datasets. However, comparing different approaches is cumbersome. The lack of existing and curated leaderboards leads to a missing global view over the research field and could inject mistrust into the results. In particular, the latest and most-used datasets in the KGQA community, LC-QuAD and QALD, miss providing central and up-to-date points of trust. In this paper, we survey and analyze a wide range of evaluation results with significant coverage of 100 publications and 98 systems from the last decade. We provide a new central and open leaderboard for any KGQA benchmark dataset as a focal point for the community - https://kgqa.github.io/leaderboard/. Our analysis highlights existing problems during the evaluation of KGQA systems. Thus, we will point to possible improvements for future evaluations.
AB - Data-driven systems need to be evaluated to establish trust in the scientific approach and its applicability. In particular, this is true for Knowledge Graph (KG) Question Answering (QA), where complex data structures are made accessible via natural-language interfaces. Evaluating the capabilities of these systems has been a driver for the community for more than ten years while establishing different KGQA benchmark datasets. However, comparing different approaches is cumbersome. The lack of existing and curated leaderboards leads to a missing global view over the research field and could inject mistrust into the results. In particular, the latest and most-used datasets in the KGQA community, LC-QuAD and QALD, miss providing central and up-to-date points of trust. In this paper, we survey and analyze a wide range of evaluation results with significant coverage of 100 publications and 98 systems from the last decade. We provide a new central and open leaderboard for any KGQA benchmark dataset as a focal point for the community - https://kgqa.github.io/leaderboard/. Our analysis highlights existing problems during the evaluation of KGQA systems. Thus, we will point to possible improvements for future evaluations.
KW - Evaluation Methodology
KW - Knowledge Graph
KW - Leaderboard
KW - Question Answering
KW - Replication Crisis
KW - Business informatics
UR - http://www.scopus.com/inward/record.url?scp=85144360908&partnerID=8YFLogxK
UR - https://www.mendeley.com/catalogue/41eb2e20-4b9b-304e-bf1e-0bfeb3b9aa51/
M3 - Article in conference proceedings
AN - SCOPUS:85144360908
T3 - 2022 Language Resources and Evaluation Conference, LREC 2022
SP - 2998
EP - 3007
BT - 2022 Language Resources and Evaluation Conference, LREC 2022
A2 - Calzolari, Nicoletta
A2 - Bechet, Frederic
A2 - Blache, Philippe
A2 - Choukri, Khalid
A2 - Cieri, Christopher
A2 - Declerck, Thierry
A2 - Goggi, Sara
A2 - Isahara, Hitoshi
A2 - Maegaard, Bente
A2 - Mariani, Joseph
A2 - Mazo, Helene
A2 - Odijk, Jan
A2 - Piperidis, Stelios
PB - European Language Resources Association (ELRA)
Y2 - 20 June 2022 through 25 June 2022
ER -