TY - CHAP

T1 - Knowledge Graph Question Answering Datasets and Their Generalizability

T2 - 45th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR 2022

AU - Jiang, Longquan

AU - Usbeck, Ricardo

N1 - Conference code: 45

PY - 2022/7/7

Y1 - 2022/7/7

N2 - Existing approaches on Question Answering over Knowledge Graphs (KGQA) have weak generalizability. That is often due to the standard i.i.d. assumption on the underlying dataset. Recently, three levels of generalization for KGQA were defined, namely i.i.d., compositional, zero-shot. We analyze 25 well-known KGQA datasets for 5 different Knowledge Graphs (KGs). We show that according to this definition many existing and online available KGQA datasets are either not suited to train a generalizable KGQA system or that the datasets are based on discontinued and out-dated KGs. Generating new datasets is a costly process and, thus, is not an alternative to smaller research groups and companies. In this work, we propose a mitigation method for re-splitting available KGQA datasets to enable their applicability to evaluate generalization, without any cost and manual effort. We test our hypothesis on three KGQA datasets, i.e., LC-QuAD, LC-QuAD 2.0 and QALD-9). Experiments on re-splitted KGQA datasets demonstrate its effectiveness towards generalizability. The code and a unified way to access 18 available datasets is online at https: //github.com/semantic-systems/KGQA-datasets as well as https: //github.com/semantic-systems/KGQA-datasets-generalization.

AB - Existing approaches on Question Answering over Knowledge Graphs (KGQA) have weak generalizability. That is often due to the standard i.i.d. assumption on the underlying dataset. Recently, three levels of generalization for KGQA were defined, namely i.i.d., compositional, zero-shot. We analyze 25 well-known KGQA datasets for 5 different Knowledge Graphs (KGs). We show that according to this definition many existing and online available KGQA datasets are either not suited to train a generalizable KGQA system or that the datasets are based on discontinued and out-dated KGs. Generating new datasets is a costly process and, thus, is not an alternative to smaller research groups and companies. In this work, we propose a mitigation method for re-splitting available KGQA datasets to enable their applicability to evaluate generalization, without any cost and manual effort. We test our hypothesis on three KGQA datasets, i.e., LC-QuAD, LC-QuAD 2.0 and QALD-9). Experiments on re-splitted KGQA datasets demonstrate its effectiveness towards generalizability. The code and a unified way to access 18 available datasets is online at https: //github.com/semantic-systems/KGQA-datasets as well as https: //github.com/semantic-systems/KGQA-datasets-generalization.

KW - benchmark

KW - evaluation

KW - generalizability

KW - generalization

KW - kgqa

KW - question answering

KW - Informatics

KW - Business informatics

UR - http://www.scopus.com/inward/record.url?scp=85135050347&partnerID=8YFLogxK

UR - https://www.mendeley.com/catalogue/b6989f93-815e-39f3-8194-cc48d3490cd6/

U2 - 10.1145/3477495.3531751

DO - 10.1145/3477495.3531751

M3 - Article in conference proceedings

AN - SCOPUS:85135050347

T3 - Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval

SP - 3209

EP - 3218

BT - SIGIR 2022 - Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

A2 - Amigo, Enrique

A2 - Castells, Pablo

A2 - Gonzalo, Julio

PB - Association for Computing Machinery, Inc

CY - New York

Y2 - 11 July 2022 through 15 July 2022

ER -