Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis

Publikation: Beiträge in SammelwerkenAufsätze in KonferenzbändenForschungbegutachtet

Standard

Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis. / Perevalov, Aleksandr; Yan, Xi; Kovriguina, Liubov et al.
2022 Language Resources and Evaluation Conference, LREC 2022. Hrsg. / Nicoletta Calzolari; Frederic Bechet; Philippe Blache; Khalid Choukri; Christopher Cieri; Thierry Declerck; Sara Goggi; Hitoshi Isahara; Bente Maegaard; Joseph Mariani; Helene Mazo; Jan Odijk; Stelios Piperidis. European Language Resources Association (ELRA), 2022. S. 2998-3007 (2022 Language Resources and Evaluation Conference, LREC 2022).

Publikation: Beiträge in SammelwerkenAufsätze in KonferenzbändenForschungbegutachtet

Harvard

Perevalov, A, Yan, X, Kovriguina, L, Jiang, L, Both, A & Usbeck, R 2022, Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis. in N Calzolari, F Bechet, P Blache, K Choukri, C Cieri, T Declerck, S Goggi, H Isahara, B Maegaard, J Mariani, H Mazo, J Odijk & S Piperidis (Hrsg.), 2022 Language Resources and Evaluation Conference, LREC 2022. 2022 Language Resources and Evaluation Conference, LREC 2022, European Language Resources Association (ELRA), S. 2998-3007, 13th International Conference on Language Resources and Evaluation Conference - LREC 2022, Marseille, Frankreich, 20.06.22. <https://aclanthology.org/2022.lrec-1.321>

APA

Perevalov, A., Yan, X., Kovriguina, L., Jiang, L., Both, A., & Usbeck, R. (2022). Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis. In N. Calzolari, F. Bechet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, J. Odijk, & S. Piperidis (Hrsg.), 2022 Language Resources and Evaluation Conference, LREC 2022 (S. 2998-3007). (2022 Language Resources and Evaluation Conference, LREC 2022). European Language Resources Association (ELRA). https://aclanthology.org/2022.lrec-1.321

Vancouver

Perevalov A, Yan X, Kovriguina L, Jiang L, Both A, Usbeck R. Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis. in Calzolari N, Bechet F, Blache P, Choukri K, Cieri C, Declerck T, Goggi S, Isahara H, Maegaard B, Mariani J, Mazo H, Odijk J, Piperidis S, Hrsg., 2022 Language Resources and Evaluation Conference, LREC 2022. European Language Resources Association (ELRA). 2022. S. 2998-3007. (2022 Language Resources and Evaluation Conference, LREC 2022).

Bibtex

@inbook{699c26425fad4ff9a856f7948345e73d,
title = "Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis",
abstract = "Data-driven systems need to be evaluated to establish trust in the scientific approach and its applicability. In particular, this is true for Knowledge Graph (KG) Question Answering (QA), where complex data structures are made accessible via natural-language interfaces. Evaluating the capabilities of these systems has been a driver for the community for more than ten years while establishing different KGQA benchmark datasets. However, comparing different approaches is cumbersome. The lack of existing and curated leaderboards leads to a missing global view over the research field and could inject mistrust into the results. In particular, the latest and most-used datasets in the KGQA community, LC-QuAD and QALD, miss providing central and up-to-date points of trust. In this paper, we survey and analyze a wide range of evaluation results with significant coverage of 100 publications and 98 systems from the last decade. We provide a new central and open leaderboard for any KGQA benchmark dataset as a focal point for the community - https://kgqa.github.io/leaderboard/. Our analysis highlights existing problems during the evaluation of KGQA systems. Thus, we will point to possible improvements for future evaluations.",
keywords = "Evaluation Methodology, Knowledge Graph, Leaderboard, Question Answering, Replication Crisis, Business informatics",
author = "Aleksandr Perevalov and Xi Yan and Liubov Kovriguina and Longquan Jiang and Andreas Both and Ricardo Usbeck",
note = "Publisher Copyright: {\textcopyright} European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0.; 13th International Conference on Language Resources and Evaluation Conference - LREC 2022 : Identify, Describe and Share your LRs!, LREC 2022 ; Conference date: 20-06-2022 Through 25-06-2022",
year = "2022",
language = "English",
series = "2022 Language Resources and Evaluation Conference, LREC 2022",
publisher = "European Language Resources Association (ELRA)",
pages = "2998--3007",
editor = "Nicoletta Calzolari and Frederic Bechet and Philippe Blache and Khalid Choukri and Christopher Cieri and Thierry Declerck and Sara Goggi and Hitoshi Isahara and Bente Maegaard and Joseph Mariani and Helene Mazo and Jan Odijk and Stelios Piperidis",
booktitle = "2022 Language Resources and Evaluation Conference, LREC 2022",
address = "Luxembourg",
url = "https://lrec2022.lrec-conf.org/en/",

}

RIS

TY - CHAP

T1 - Knowledge Graph Question Answering Leaderboard

T2 - 13th International Conference on Language Resources and Evaluation Conference - LREC 2022

AU - Perevalov, Aleksandr

AU - Yan, Xi

AU - Kovriguina, Liubov

AU - Jiang, Longquan

AU - Both, Andreas

AU - Usbeck, Ricardo

N1 - Conference code: 13

PY - 2022

Y1 - 2022

N2 - Data-driven systems need to be evaluated to establish trust in the scientific approach and its applicability. In particular, this is true for Knowledge Graph (KG) Question Answering (QA), where complex data structures are made accessible via natural-language interfaces. Evaluating the capabilities of these systems has been a driver for the community for more than ten years while establishing different KGQA benchmark datasets. However, comparing different approaches is cumbersome. The lack of existing and curated leaderboards leads to a missing global view over the research field and could inject mistrust into the results. In particular, the latest and most-used datasets in the KGQA community, LC-QuAD and QALD, miss providing central and up-to-date points of trust. In this paper, we survey and analyze a wide range of evaluation results with significant coverage of 100 publications and 98 systems from the last decade. We provide a new central and open leaderboard for any KGQA benchmark dataset as a focal point for the community - https://kgqa.github.io/leaderboard/. Our analysis highlights existing problems during the evaluation of KGQA systems. Thus, we will point to possible improvements for future evaluations.

AB - Data-driven systems need to be evaluated to establish trust in the scientific approach and its applicability. In particular, this is true for Knowledge Graph (KG) Question Answering (QA), where complex data structures are made accessible via natural-language interfaces. Evaluating the capabilities of these systems has been a driver for the community for more than ten years while establishing different KGQA benchmark datasets. However, comparing different approaches is cumbersome. The lack of existing and curated leaderboards leads to a missing global view over the research field and could inject mistrust into the results. In particular, the latest and most-used datasets in the KGQA community, LC-QuAD and QALD, miss providing central and up-to-date points of trust. In this paper, we survey and analyze a wide range of evaluation results with significant coverage of 100 publications and 98 systems from the last decade. We provide a new central and open leaderboard for any KGQA benchmark dataset as a focal point for the community - https://kgqa.github.io/leaderboard/. Our analysis highlights existing problems during the evaluation of KGQA systems. Thus, we will point to possible improvements for future evaluations.

KW - Evaluation Methodology

KW - Knowledge Graph

KW - Leaderboard

KW - Question Answering

KW - Replication Crisis

KW - Business informatics

UR - http://www.scopus.com/inward/record.url?scp=85144360908&partnerID=8YFLogxK

UR - https://www.mendeley.com/catalogue/41eb2e20-4b9b-304e-bf1e-0bfeb3b9aa51/

M3 - Article in conference proceedings

AN - SCOPUS:85144360908

T3 - 2022 Language Resources and Evaluation Conference, LREC 2022

SP - 2998

EP - 3007

BT - 2022 Language Resources and Evaluation Conference, LREC 2022

A2 - Calzolari, Nicoletta

A2 - Bechet, Frederic

A2 - Blache, Philippe

A2 - Choukri, Khalid

A2 - Cieri, Christopher

A2 - Declerck, Thierry

A2 - Goggi, Sara

A2 - Isahara, Hitoshi

A2 - Maegaard, Bente

A2 - Mariani, Joseph

A2 - Mazo, Helene

A2 - Odijk, Jan

A2 - Piperidis, Stelios

PB - European Language Resources Association (ELRA)

Y2 - 20 June 2022 through 25 June 2022

ER -

Links