Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

  • Aleksandr Perevalov
  • Xi Yan
  • Liubov Kovriguina
  • Longquan Jiang
  • Andreas Both
  • Ricardo Usbeck

Data-driven systems need to be evaluated to establish trust in the scientific approach and its applicability. In particular, this is true for Knowledge Graph (KG) Question Answering (QA), where complex data structures are made accessible via natural-language interfaces. Evaluating the capabilities of these systems has been a driver for the community for more than ten years while establishing different KGQA benchmark datasets. However, comparing different approaches is cumbersome. The lack of existing and curated leaderboards leads to a missing global view over the research field and could inject mistrust into the results. In particular, the latest and most-used datasets in the KGQA community, LC-QuAD and QALD, miss providing central and up-to-date points of trust. In this paper, we survey and analyze a wide range of evaluation results with significant coverage of 100 publications and 98 systems from the last decade. We provide a new central and open leaderboard for any KGQA benchmark dataset as a focal point for the community - https://kgqa.github.io/leaderboard/. Our analysis highlights existing problems during the evaluation of KGQA systems. Thus, we will point to possible improvements for future evaluations.

Original languageEnglish
Title of host publication2022 Language Resources and Evaluation Conference, LREC 2022
EditorsNicoletta Calzolari, Frederic Bechet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Helene Mazo, Jan Odijk, Stelios Piperidis
Number of pages10
PublisherEuropean Language Resources Association (ELRA)
Publication date2022
Pages2998-3007
ISBN (electronic)9791095546726
Publication statusPublished - 2022
Externally publishedYes
Event13th International Conference on Language Resources and Evaluation Conference - LREC 2022: Identify, Describe and Share your LRs! - Palais du Pharo, Marseille, France
Duration: 20.06.202225.06.2022
Conference number: 13
https://lrec2022.lrec-conf.org/en/

Bibliographical note

Publisher Copyright:
© European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0.

    Research areas

  • Evaluation Methodology, Knowledge Graph, Leaderboard, Question Answering, Replication Crisis
  • Business informatics

Links

Recently viewed

Publications

  1. Geochemical Assessment of Sediment Quality Using Multivariate Statistical Analysis of Ennore Creek, North of Chennai, SE Coast of India.
  2. Geistessterben
  3. From Allegiant to Assertive Citizens
  4. Direct negative density-dependence in a pond-breeding frog population
  5. Nachwuchsförderung in Drittmittelprojekten - "untiteled" oder "Writer"?
  6. Back to the roots and back to the future
  7. Transport in dynamical astronomy and multibody problems
  8. Digital Transformation of Interdisciplinary Engineering Education
  9. Social and dimensional comparison effects on math and reading self-concepts of elementary school children
  10. On the notion of ecological justice
  11. What we mean when we talk about freedom – The KOMFOR study
  12. Gamification as twenty-first-century ideology
  13. Publicum
  14. Notting Hill Gate 4
  15. Feedstocks and analysis
  16. Gone with the wind?
  17. Framing climate uncertainty
  18. Comparison of PSA-specific CD8+ CTL responses and antitumor immunity generated by plasmid DNA vaccines encoding PSA-HSP chimeric proteins
  19. Ambient Assited Energy Management
  20. A mixed-method approach to post-retirement career planning
  21. Assessing the added value of the recent declaration on unregulated fishing for sustainable governance of the central Arctic Ocean
  22. Gamification
  23. Bildungsstandards ja - aber wie?
  24. Experimental and theoretical investigation of the microstructural evolution in aluminium alloys during extrusion
  25. We build this city on rocks and (feminist) code
  26. Automation in Clinical Laboratories
  27. A review of ecological gradient research in the Tropics