GANDR - Georelating Dataset, Metrics, and Evaluation
Research output: Contributions to collected editions/works › Article in conference proceedings › Research › peer-review
Standard
GeoAI '25: Proceedings of the 8th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery. ed. / Shawn Newsam; Lexie Yang. New York, NY, USA: Association for Computing Machinery, 2025. p. 61-71 (GeoAI '25).
Research output: Contributions to collected editions/works › Article in conference proceedings › Research › peer-review
Harvard
APA
Vancouver
Bibtex
}
RIS
TY - CHAP
T1 - GANDR - Georelating Dataset, Metrics, and Evaluation
AU - Moltzen, Kai
AU - Usbeck, Ricardo
PY - 2025/12/19
Y1 - 2025/12/19
N2 - Georelating has been introduced to learn geospatial representations of events from textual reports, which requires the interpretation of spatial relations. To foster the development and evaluation of Georelating systems, we construct the silver-standard Georelating Annotated Natural Disaster Reports dataset GANDR and benchmark our LLM agent architecture as a baseline (areal F1 = 0.609, fuzzy cell match score = 0.833) for this new task.GANDR comprises synthetic disaster reports referencing 1,000 US and 1,000 EU cities, annotated with Discrete Global Grid System (DGGS) cells for efficient geospatial integration. We propose a set of five complementary metrics capitalizing on the DGGS annotations for efficient and comprehensive evaluation.Analysis reveals the potential of reasoning LLMs integrated with geographical knowledge bases to address variation across spatial relations such as (inter-)cardinal directions. We highlight the estimation of the impact area's size as a key challenge of Georelating.
AB - Georelating has been introduced to learn geospatial representations of events from textual reports, which requires the interpretation of spatial relations. To foster the development and evaluation of Georelating systems, we construct the silver-standard Georelating Annotated Natural Disaster Reports dataset GANDR and benchmark our LLM agent architecture as a baseline (areal F1 = 0.609, fuzzy cell match score = 0.833) for this new task.GANDR comprises synthetic disaster reports referencing 1,000 US and 1,000 EU cities, annotated with Discrete Global Grid System (DGGS) cells for efficient geospatial integration. We propose a set of five complementary metrics capitalizing on the DGGS annotations for efficient and comprehensive evaluation.Analysis reveals the potential of reasoning LLMs integrated with geographical knowledge bases to address variation across spatial relations such as (inter-)cardinal directions. We highlight the estimation of the impact area's size as a key challenge of Georelating.
KW - georelating
KW - LLMs
KW - spatial reasoning
KW - DGGS
KW - dataset
KW - metrics
KW - Informatics
U2 - 10.1145/3764912.3770819
DO - 10.1145/3764912.3770819
M3 - Article in conference proceedings
SN - 9798400721793
T3 - GeoAI '25
SP - 61
EP - 71
BT - GeoAI '25: Proceedings of the 8th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery
A2 - Newsam, Shawn
A2 - Yang, Lexie
PB - Association for Computing Machinery
CY - New York, NY, USA
ER -