ShortPathQA: A Dataset for Controllable Fusion of Large Language Models with Knowledge Graphs
Abstract: In this work, we release the Shortest Path subgraph Question Answering (ShortPathQA) dataset, the first dataset that provides textual questions with pre-computed relevant subgraphs retrieved from the Wikidata Knowledge Graph (KG), standardizing the evaluation framework for Knowledge Graph Question Answering (KGQA). For this purpose, we utilize the Mintaka dataset for both training and testing and additionally create a manual question-answering subset for testing. Our baseline experiments with both supervised approaches and unsupervised Large Language Model (LLM) inference indicate that even a simplified KGQA formulation with given KG subgraphs and candidate answers remains challenging. Our analysis has shown that LLMs are unable to correctly process and utilize graph data structures without detailed prompt engineering or model tuning. This limitation highlights the need for the creation of this dataset as a training ground for the development of methods that enable LLMs to work more effectively with graph data.
Show BibTeX
@inproceedings{DBLP:conf/nldb/SalnikovSNUKMBHJAYTUP25,
author = {Mikhail Salnikov and
Andrey Sakhovskiy and
Irina Nikishina and
Aida Usmanova and
Angelie Kraft and
Cedric M{\"{o}}ller and
Debayan Banerjee and
Junbo Huang and
Longquan Jiang and
Rana Abdullah and
Xi Yan and
Elena Tutubalina and
Ricardo Usbeck and
Alexander Panchenko},
editor = {Ryutaro Ichise},
title = {ShortPathQA: {A} Dataset for Controllable Fusion of Large Language
Models with Knowledge Graphs},
booktitle = {Natural Language Processing and Information Systems - 30th International
Conference on Applications of Natural Language to Information Systems,
{NLDB} 2025, Kanazawa, Japan, July 4-6, 2025, Proceedings, Part {I}},
series = {Lecture Notes in Computer Science},
volume = {15836},
pages = {95--110},
publisher = {Springer},
year = {2025},
url = {https://doi.org/10.1007/978-3-031-97141-9\_7},
doi = {10.1007/978-3-031-97141-9\_7},
timestamp = {Sat, 18 Jul 2026 13:05:48 +0200},
biburl = {https://dblp.org/rec/conf/nldb/SalnikovSNUKMBHJAYTUP25.bib},
bibsource = {dblp computer science bibliography, https://dblp.org}
}