Gerbil – Benchmarking named entity recognition and linking consistently

Research output: Journal contributionsJournal articlesResearchpeer-review

Standard

Gerbil – Benchmarking named entity recognition and linking consistently. / Röder, Michael; Usbeck, Ricardo; Ngonga Ngomo, Axel Cyrille.
In: Semantic Web, Vol. 9, No. 5, 2018.

Research output: Journal contributionsJournal articlesResearchpeer-review

Harvard

APA

Vancouver

Röder M, Usbeck R, Ngonga Ngomo AC. Gerbil – Benchmarking named entity recognition and linking consistently. Semantic Web. 2018;9(5). doi: 10.3233/sw-170286

Bibtex

@article{da05985e99fb43258bf73e7e9d3009e7,
title = "Gerbil – Benchmarking named entity recognition and linking consistently",
abstract = "The ability to compare systems from the same domain is of central importance for their introduction into complex applications. In the domains of named entity recognition and entity linking, the large number of systems and their orthogonal evaluation w.r.t. measures and datasets has led to an unclear landscape regarding the abilities and weaknesses of the different approaches. We present GERBIL—an improved platform for repeatable, storable and citable semantic annotation experiments—and its extension since being release. GERBIL has narrowed this evaluation gap by generating concise, archivable, human- and machine-readable experiments, analytics and diagnostics. The rationale behind our framework is to provide developers, end users and researchers with easy-to-use interfaces that allow for the agile, fine-grained and uniform evaluation of annotation tools on multiple datasets. By these means, we aim to ensure that both tool developers and end users can derive meaningful insights into the extension, integration and use of annotation applications. In particular, GERBIL provides comparable results to tool developers, simplifying the discovery of strengths and weaknesses of their implementations with respect to the state-of-the-art. With the permanent experiment URIs provided by our framework, we ensure the reproducibility and archiving of evaluation results. Moreover, the framework generates data in a machine-processable format, allowing for the efficient querying and post-processing of evaluation results. Additionally, the tool diagnostics provided by GERBIL provide insights into the areas where tools need further refinement, thus allowing developers to create an informed agenda for extensions and end users to detect the right tools for their purposes. Finally, we implemented additional types of experiments including entity typing. GERBIL aims to become a focal point for the state-of-the-art, driving the research agenda of the community by presenting comparable objective evaluation results. Furthermore, we tackle the central problem of the evaluation of entity linking, i.e., we answer the question of how an evaluation algorithm can compare two URIs to each other without being bound to a specific knowledge base. Our approach to this problem opens a way to address the deprecation of URIs of existing gold standards for named entity recognition and entity linking, a feature which is currently not supported by the state-of-the-art. We derived the importance of this feature from usage and dataset requirements collected from the GERBIL user community, which has already carried out more than 24.000 single evaluations using our framework. Through the resulting updates, GERBIL now supports 8 tasks, 46 datasets and 20 systems.",
keywords = "Archivability, Benchmarking framework, Disambiguation, Linking, Named entity recognition, Reusability, Semantic entity annotation system, Informatics, Business informatics",
author = "Michael R{\"o}der and Ricardo Usbeck and {Ngonga Ngomo}, {Axel Cyrille}",
note = "This work was supported by the German Federal Ministry of Education and Research under the project number 03WKCJ4D and the Eurostars projects DIESEL (E!9367) and QAMEL (E!9725) as well as the European Union{\textquoteright}s H2020 research and innovation action HOBBIT under the Grant Agreement number 688227. Publisher Copyright: {\textcopyright} 2018 IOS Press. All rights reserved.",
year = "2018",
doi = "10.3233/sw-170286",
language = "English",
volume = "9",
journal = "Semantic Web",
issn = "1570-0844",
publisher = "SAGE Publications Inc.",
number = "5",

}

RIS

TY - JOUR

T1 - Gerbil – Benchmarking named entity recognition and linking consistently

AU - Röder, Michael

AU - Usbeck, Ricardo

AU - Ngonga Ngomo, Axel Cyrille

N1 - This work was supported by the German Federal Ministry of Education and Research under the project number 03WKCJ4D and the Eurostars projects DIESEL (E!9367) and QAMEL (E!9725) as well as the European Union’s H2020 research and innovation action HOBBIT under the Grant Agreement number 688227. Publisher Copyright: © 2018 IOS Press. All rights reserved.

PY - 2018

Y1 - 2018

N2 - The ability to compare systems from the same domain is of central importance for their introduction into complex applications. In the domains of named entity recognition and entity linking, the large number of systems and their orthogonal evaluation w.r.t. measures and datasets has led to an unclear landscape regarding the abilities and weaknesses of the different approaches. We present GERBIL—an improved platform for repeatable, storable and citable semantic annotation experiments—and its extension since being release. GERBIL has narrowed this evaluation gap by generating concise, archivable, human- and machine-readable experiments, analytics and diagnostics. The rationale behind our framework is to provide developers, end users and researchers with easy-to-use interfaces that allow for the agile, fine-grained and uniform evaluation of annotation tools on multiple datasets. By these means, we aim to ensure that both tool developers and end users can derive meaningful insights into the extension, integration and use of annotation applications. In particular, GERBIL provides comparable results to tool developers, simplifying the discovery of strengths and weaknesses of their implementations with respect to the state-of-the-art. With the permanent experiment URIs provided by our framework, we ensure the reproducibility and archiving of evaluation results. Moreover, the framework generates data in a machine-processable format, allowing for the efficient querying and post-processing of evaluation results. Additionally, the tool diagnostics provided by GERBIL provide insights into the areas where tools need further refinement, thus allowing developers to create an informed agenda for extensions and end users to detect the right tools for their purposes. Finally, we implemented additional types of experiments including entity typing. GERBIL aims to become a focal point for the state-of-the-art, driving the research agenda of the community by presenting comparable objective evaluation results. Furthermore, we tackle the central problem of the evaluation of entity linking, i.e., we answer the question of how an evaluation algorithm can compare two URIs to each other without being bound to a specific knowledge base. Our approach to this problem opens a way to address the deprecation of URIs of existing gold standards for named entity recognition and entity linking, a feature which is currently not supported by the state-of-the-art. We derived the importance of this feature from usage and dataset requirements collected from the GERBIL user community, which has already carried out more than 24.000 single evaluations using our framework. Through the resulting updates, GERBIL now supports 8 tasks, 46 datasets and 20 systems.

AB - The ability to compare systems from the same domain is of central importance for their introduction into complex applications. In the domains of named entity recognition and entity linking, the large number of systems and their orthogonal evaluation w.r.t. measures and datasets has led to an unclear landscape regarding the abilities and weaknesses of the different approaches. We present GERBIL—an improved platform for repeatable, storable and citable semantic annotation experiments—and its extension since being release. GERBIL has narrowed this evaluation gap by generating concise, archivable, human- and machine-readable experiments, analytics and diagnostics. The rationale behind our framework is to provide developers, end users and researchers with easy-to-use interfaces that allow for the agile, fine-grained and uniform evaluation of annotation tools on multiple datasets. By these means, we aim to ensure that both tool developers and end users can derive meaningful insights into the extension, integration and use of annotation applications. In particular, GERBIL provides comparable results to tool developers, simplifying the discovery of strengths and weaknesses of their implementations with respect to the state-of-the-art. With the permanent experiment URIs provided by our framework, we ensure the reproducibility and archiving of evaluation results. Moreover, the framework generates data in a machine-processable format, allowing for the efficient querying and post-processing of evaluation results. Additionally, the tool diagnostics provided by GERBIL provide insights into the areas where tools need further refinement, thus allowing developers to create an informed agenda for extensions and end users to detect the right tools for their purposes. Finally, we implemented additional types of experiments including entity typing. GERBIL aims to become a focal point for the state-of-the-art, driving the research agenda of the community by presenting comparable objective evaluation results. Furthermore, we tackle the central problem of the evaluation of entity linking, i.e., we answer the question of how an evaluation algorithm can compare two URIs to each other without being bound to a specific knowledge base. Our approach to this problem opens a way to address the deprecation of URIs of existing gold standards for named entity recognition and entity linking, a feature which is currently not supported by the state-of-the-art. We derived the importance of this feature from usage and dataset requirements collected from the GERBIL user community, which has already carried out more than 24.000 single evaluations using our framework. Through the resulting updates, GERBIL now supports 8 tasks, 46 datasets and 20 systems.

KW - Archivability

KW - Benchmarking framework

KW - Disambiguation

KW - Linking

KW - Named entity recognition

KW - Reusability

KW - Semantic entity annotation system

KW - Informatics

KW - Business informatics

UR - http://www.scopus.com/inward/record.url?scp=85066125676&partnerID=8YFLogxK

U2 - 10.3233/sw-170286

DO - 10.3233/sw-170286

M3 - Journal articles

AN - SCOPUS:85066125676

VL - 9

JO - Semantic Web

JF - Semantic Web

SN - 1570-0844

IS - 5

ER -

DOI

Recently viewed

Researchers

  1. Nele Meyer

Publications

  1. Reliability and Validity of Assessing User Satisfaction With Web-Based Health Interventions
  2. Comparison of modeling approaches based on the microstructure of thermally sprayed coatings
  3. Modeling Efficient Grounding in Chat-based CSCL
  4. Joint Proceedings of Scholarly QALD 2023 and SemREC 2023 co-located with 22nd International Semantic Web Conference ISWC 2023
  5. Traffic Life: Temporal Dynamics and Regulatory Dimensions in Agent-Based Transport Simulations
  6. The use of the online Inverted Classroom Model for digital teaching with gamification in medical studies
  7. Extraterritorial Asylum Processing. The Libya-Niger Emergency Transit Mechanism
  8. Non-native populations of an invasive tree outperform their native conspecifics
  9. Using work values to predict post-retirement work intentions
  10. "If you like something, you want it to develop."
  11. Investigating quality mathematics teaching - the DISUM project
  12. Predator assemblage structure and temporal variability of species richness and abundance in forests of high tree diversity
  13. y-Randomization and its variants in QSPR/QSAR
  14. Modeling Self-Organization
  15. The dependency of the banks’ assets and liabilities
  16. Payments for ecosystem services – for efficiency and for equity?
  17. Understanding role models for change
  18. Computer-Kriegs-Spiele oder: eine Kultur der Gewalt
  19. A new didactic approach in Engineering Education for conceptual understanding of Euler's Formula
  20. Numerical responses of saproxylic beetles to rapid increases in dead wood availability following geometrid moth outbreaks in sub-arctic mountain birch forest
  21. Credit Constraints and Exports
  22. Erosion modelling designed for water quality simulation
  23. Lernende in der Hauptschule
  24. Mentoring in International Assignments
  25. An idea and a person whose time had come
  26. Update wurde nicht ausgeführt
  27. Canopy functional trait variation across Earth’s tropical forests