Knowledge Graph Question Answering Datasets and Their Generalizability: Are They Enough for Future Research?

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

Existing approaches on Question Answering over Knowledge Graphs (KGQA) have weak generalizability. That is often due to the standard i.i.d. assumption on the underlying dataset. Recently, three levels of generalization for KGQA were defined, namely i.i.d., compositional, zero-shot. We analyze 25 well-known KGQA datasets for 5 different Knowledge Graphs (KGs). We show that according to this definition many existing and online available KGQA datasets are either not suited to train a generalizable KGQA system or that the datasets are based on discontinued and out-dated KGs. Generating new datasets is a costly process and, thus, is not an alternative to smaller research groups and companies. In this work, we propose a mitigation method for re-splitting available KGQA datasets to enable their applicability to evaluate generalization, without any cost and manual effort. We test our hypothesis on three KGQA datasets, i.e., LC-QuAD, LC-QuAD 2.0 and QALD-9). Experiments on re-splitted KGQA datasets demonstrate its effectiveness towards generalizability. The code and a unified way to access 18 available datasets is online at https: //github.com/semantic-systems/KGQA-datasets as well as https: //github.com/semantic-systems/KGQA-datasets-generalization.

Original languageEnglish
Title of host publicationSIGIR 2022 - Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval
EditorsEnrique Amigo, Pablo Castells, Julio Gonzalo
Number of pages10
Place of PublicationNew York
PublisherAssociation for Computing Machinery, Inc
Publication date07.07.2022
Pages3209-3218
ISBN (electronic)9781450387323
DOIs
Publication statusPublished - 07.07.2022
Externally publishedYes
Event45th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR 2022 - Online + Círculo de Bellas Artes (Circle of Beaux Arts), Madrid, Spain
Duration: 11.07.202215.07.2022
Conference number: 45
https://sigir.org/sigir2022/

Bibliographical note

Publisher Copyright:
© 2022 ACM.

Recently viewed

Researchers

  1. Inge Nehring

Projects

  1. The art markets

Publications

  1. An optimal minimum phase approximating PD regulator for robust control of a throttle plate
  2. Circular Scanning Resolution Improvement by its Velocity Close Loop Control
  3. Understanding and managing post-acquisition integration as change process
  4. Shared mobility business models
  5. Dispute and morality in the perception of societal risks: extending the psychometric model
  6. An Off-the-shelf Approach to Authorship Attribution
  7. The Inada conditions for material resource inputs reconsidered
  8. Leaf Nutritional Content, Tree Richness, and Season Shape the Caterpillar Functional Trait Composition Hosted by Trees
  9. Algorithmisches Management
  10. Comparative effectiveness of guided internet-based stress management training versus established in-person group training in employees – study protocol for a pragmatic, randomized, non-inferiority trial
  11. Self-Regulated Learning with Expository Texts as a Competence
  12. Transfer of metacognitive skills in self-regulated learning
  13. Self-Compassion as a Facet of Neuroticism? A Reply to the Comments of Neff, Tóth-Király, and Colosimo (2018)
  14. Performance pay sensitivity: Do top management incentives align with shareholder value creation?
  15. Understanding spam
  16. From deforestation to blossom
  17. Smarte Anpassung von Presslinienparametern
  18. Native vegetation cover thresholds associated with species responses
  19. The challenge of managing multiple species at multiple scales
  20. Comparison through conversation
  21. Microstructure and hardness evolution of laser metal deposited AA5087 wall-structures
  22. “Normality” Revisited: Fieldwork and Family
  23. Virtual Voting in RFMOs
  24. The Third Image