TY - CHAP

T1 - Knowledge Graph Question Answering Leaderboard

T2 - 13th International Conference on Language Resources and Evaluation Conference - LREC 2022

AU - Perevalov, Aleksandr

AU - Yan, Xi

AU - Kovriguina, Liubov

AU - Jiang, Longquan

AU - Both, Andreas

AU - Usbeck, Ricardo

N1 - Conference code: 13

PY - 2022

Y1 - 2022

N2 - Data-driven systems need to be evaluated to establish trust in the scientific approach and its applicability. In particular, this is true for Knowledge Graph (KG) Question Answering (QA), where complex data structures are made accessible via natural-language interfaces. Evaluating the capabilities of these systems has been a driver for the community for more than ten years while establishing different KGQA benchmark datasets. However, comparing different approaches is cumbersome. The lack of existing and curated leaderboards leads to a missing global view over the research field and could inject mistrust into the results. In particular, the latest and most-used datasets in the KGQA community, LC-QuAD and QALD, miss providing central and up-to-date points of trust. In this paper, we survey and analyze a wide range of evaluation results with significant coverage of 100 publications and 98 systems from the last decade. We provide a new central and open leaderboard for any KGQA benchmark dataset as a focal point for the community - https://kgqa.github.io/leaderboard/. Our analysis highlights existing problems during the evaluation of KGQA systems. Thus, we will point to possible improvements for future evaluations.

AB - Data-driven systems need to be evaluated to establish trust in the scientific approach and its applicability. In particular, this is true for Knowledge Graph (KG) Question Answering (QA), where complex data structures are made accessible via natural-language interfaces. Evaluating the capabilities of these systems has been a driver for the community for more than ten years while establishing different KGQA benchmark datasets. However, comparing different approaches is cumbersome. The lack of existing and curated leaderboards leads to a missing global view over the research field and could inject mistrust into the results. In particular, the latest and most-used datasets in the KGQA community, LC-QuAD and QALD, miss providing central and up-to-date points of trust. In this paper, we survey and analyze a wide range of evaluation results with significant coverage of 100 publications and 98 systems from the last decade. We provide a new central and open leaderboard for any KGQA benchmark dataset as a focal point for the community - https://kgqa.github.io/leaderboard/. Our analysis highlights existing problems during the evaluation of KGQA systems. Thus, we will point to possible improvements for future evaluations.

KW - Evaluation Methodology

KW - Knowledge Graph

KW - Leaderboard

KW - Question Answering

KW - Replication Crisis

KW - Business informatics

UR - http://www.scopus.com/inward/record.url?scp=85144360908&partnerID=8YFLogxK

UR - https://www.mendeley.com/catalogue/41eb2e20-4b9b-304e-bf1e-0bfeb3b9aa51/

M3 - Article in conference proceedings

AN - SCOPUS:85144360908

T3 - 2022 Language Resources and Evaluation Conference, LREC 2022

SP - 2998

EP - 3007

BT - 2022 Language Resources and Evaluation Conference, LREC 2022

A2 - Calzolari, Nicoletta

A2 - Bechet, Frederic

A2 - Blache, Philippe

A2 - Choukri, Khalid

A2 - Cieri, Christopher

A2 - Declerck, Thierry

A2 - Goggi, Sara

A2 - Isahara, Hitoshi

A2 - Maegaard, Bente

A2 - Mariani, Joseph

A2 - Mazo, Helene

A2 - Odijk, Jan

A2 - Piperidis, Stelios

PB - European Language Resources Association (ELRA)

Y2 - 20 June 2022 through 25 June 2022

ER -