ShortPathQA: A Dataset for Controllable Fusion of Large Language Models with Knowledge Graphs

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

In this work, we release the Shortest Path subgraph Question Answering (ShortPathQA) dataset, the first dataset that provides textual questions with pre-computed relevant subgraphs retrieved from the Wikidata Knowledge Graph (KG), standardizing the evaluation framework for Knowledge Graph Question Answering (KGQA). For this purpose, we utilize the Mintaka dataset for both training and testing and additionally create a manual question-answering subset for testing. Our baseline experiments with both supervised approaches and unsupervised Large Language Model (LLM) inference indicate that even a simplified KGQA formulation with given KG subgraphs and candidate answers remains challenging. Our analysis has shown that LLMs are unable to correctly process and utilize graph data structures without detailed prompt engineering or model tuning. This limitation highlights the need for the creation of this dataset as a training ground for the development of methods that enable LLMs to work more effectively with graph data.

Original languageEnglish
Title of host publicationNatural Language Processing and Information Systems : 30th International Conference on Applications of Natural Language to Information Systems, NLDB 2025, Proceedings
EditorsRyutaro Ichise
Number of pages16
PublisherSpringer Science and Business Media Deutschland
Publication date2026
Pages95-110
ISBN (print)978-3-031-97140-2
ISBN (electronic)978-3-031-97141-9
DOIs
Publication statusE-pub ahead of print - 01.07.2025
Event30th International Conference on Natural Language and Information Systems - NLDB 2025 - Kanazawa, Japan
Duration: 04.07.202506.07.2025
Conference number: 30

Bibliographical note

Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2026.

    Research areas

  • KGQA, Knowledge graphs, NLP, Question answering
  • Informatics