Knowledge Graph Question Answering Datasets and Their Generalizability: Are They Enough for Future Research?

Publikation: Beiträge in SammelwerkenAufsätze in KonferenzbändenForschungbegutachtet

Authors

Existing approaches on Question Answering over Knowledge Graphs (KGQA) have weak generalizability. That is often due to the standard i.i.d. assumption on the underlying dataset. Recently, three levels of generalization for KGQA were defined, namely i.i.d., compositional, zero-shot. We analyze 25 well-known KGQA datasets for 5 different Knowledge Graphs (KGs). We show that according to this definition many existing and online available KGQA datasets are either not suited to train a generalizable KGQA system or that the datasets are based on discontinued and out-dated KGs. Generating new datasets is a costly process and, thus, is not an alternative to smaller research groups and companies. In this work, we propose a mitigation method for re-splitting available KGQA datasets to enable their applicability to evaluate generalization, without any cost and manual effort. We test our hypothesis on three KGQA datasets, i.e., LC-QuAD, LC-QuAD 2.0 and QALD-9). Experiments on re-splitted KGQA datasets demonstrate its effectiveness towards generalizability. The code and a unified way to access 18 available datasets is online at https: //github.com/semantic-systems/KGQA-datasets as well as https: //github.com/semantic-systems/KGQA-datasets-generalization.

OriginalspracheEnglisch
TitelSIGIR 2022 - Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval
HerausgeberEnrique Amigo, Pablo Castells, Julio Gonzalo
Anzahl der Seiten10
ErscheinungsortNew York
VerlagAssociation for Computing Machinery, Inc
Erscheinungsdatum06.07.2022
Seiten3209-3218
ISBN (elektronisch)9781450387323
DOIs
PublikationsstatusErschienen - 06.07.2022
Extern publiziertJa
Veranstaltung45th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR 2022 - Online + Círculo de Bellas Artes (Circle of Beaux Arts), Madrid, Spanien
Dauer: 11.07.202215.07.2022
Konferenznummer: 45
https://sigir.org/sigir2022/

Bibliographische Notiz

Funding Information:
The authors acknowledge the financial support by the Federal Ministry for Economic Affairs and Climate Action of Germany in the project CoyPu (project number 01MK21007G).

Publisher Copyright:
© 2022 ACM.

Zuletzt angesehen

Aktivitäten

  1. Mathematik und Sprache
  2. Development of PE pre-service teachers’ teaching performance during a long-term internship - A mixed-methods analysis of classroom videos and the written self-reflections of the PE pre-service teachers
  3. 15th International Conference on Sensors and Measurement Technology - SENSOR 2011
  4. Tagung "Mathe für alle" 2012
  5. Tourismuspolitischer Dialog - 2013
  6. Deferred Compensation Schemes, Fairness Concerns, and Employment of Older Workers
  7. Research Priorities in Light of the Future Global Action Programme on Education for Sustainable Development
  8. Intercultural Relations in Practice 2017
  9. "Integration on the ground" - Assumptions and Consequences of Neighbourhood Strategies
  10. 19th Annual SemFest
  11. The Predictive Power of Social Media Sentiment for Short-Term Stock Movements
  12. Hyperkult XXV - 2015
  13. Education for Sustainable Adaptation to Climate Change in Tourism Branch. Using Scenarios for Knowledge Transfer
  14. CfP Sektionstagung 2017
  15. Towards a New Aesthetic Paradigm
  16. Manet und die Revolution des Impressionismus
  17. Identifying Global Challenges for Future Tourism and Tourism Management
  18. Podiumsdiskussion zu »Heuschrecken«, Rimini Protokoll
  19. The effect of professional development on teachers’ PCK, on beliefs and on the quality of teaching
  20. Partisanship and Political Affection. The normative and epistemic functions of partisan discourse
  21. Università di Macerata
  22. Foucault trifft Latour
  23. Between Path Dependency and Tourism Area Life Cycle: Cultural Assets in the Light of Spa Re-development in East Germany
  24. Wien Depot: Podiumsdiskussion
  25. Enterprise Resource Planning Systems: New transparency, new opacity
  26. Karlstad Universität
  27. East German Spas: between path dependency and tourism life cycle
  28. Methods for Ph.D.
  29. Die Symbolgrafik als dritter Weg
  30. How important are SDGs for teacher educators? – Engage educators in a global landscape: Mapping the terrain with a case study of a pilot professional development program from Ethiopia
  31. Creating Space for Academic Feedom: Progressive Liberal Education in a German Public University