Gerbil – Benchmarking named entity recognition and linking consistently

Publikation: Beiträge in ZeitschriftenZeitschriftenaufsätzeForschungbegutachtet

Authors

The ability to compare systems from the same domain is of central importance for their introduction into complex applications. In the domains of named entity recognition and entity linking, the large number of systems and their orthogonal evaluation w.r.t. measures and datasets has led to an unclear landscape regarding the abilities and weaknesses of the different approaches. We present GERBIL—an improved platform for repeatable, storable and citable semantic annotation experiments—and its extension since being release. GERBIL has narrowed this evaluation gap by generating concise, archivable, human- and machine-readable experiments, analytics and diagnostics. The rationale behind our framework is to provide developers, end users and researchers with easy-to-use interfaces that allow for the agile, fine-grained and uniform evaluation of annotation tools on multiple datasets. By these means, we aim to ensure that both tool developers and end users can derive meaningful insights into the extension, integration and use of annotation applications. In particular, GERBIL provides comparable results to tool developers, simplifying the discovery of strengths and weaknesses of their implementations with respect to the state-of-the-art. With the permanent experiment URIs provided by our framework, we ensure the reproducibility and archiving of evaluation results. Moreover, the framework generates data in a machine-processable format, allowing for the efficient querying and post-processing of evaluation results. Additionally, the tool diagnostics provided by GERBIL provide insights into the areas where tools need further refinement, thus allowing developers to create an informed agenda for extensions and end users to detect the right tools for their purposes. Finally, we implemented additional types of experiments including entity typing. GERBIL aims to become a focal point for the state-of-the-art, driving the research agenda of the community by presenting comparable objective evaluation results. Furthermore, we tackle the central problem of the evaluation of entity linking, i.e., we answer the question of how an evaluation algorithm can compare two URIs to each other without being bound to a specific knowledge base. Our approach to this problem opens a way to address the deprecation of URIs of existing gold standards for named entity recognition and entity linking, a feature which is currently not supported by the state-of-the-art. We derived the importance of this feature from usage and dataset requirements collected from the GERBIL user community, which has already carried out more than 24.000 single evaluations using our framework. Through the resulting updates, GERBIL now supports 8 tasks, 46 datasets and 20 systems.

OriginalspracheEnglisch
ZeitschriftSemantic Web
Jahrgang9
Ausgabenummer5
ISSN1570-0844
DOIs
PublikationsstatusErschienen - 2018
Extern publiziertJa

Bibliographische Notiz

Funding Information:
This work was supported by the German Federal Ministry of Education and Research under the project number 03WKCJ4D and the Eurostars projects DIESEL (E!9367) and QAMEL (E!9725) as well as the European Union’s H2020 research and innovation action HOBBIT under the Grant Agreement number 688227.

Funding Information:
Acknowledgments. This work was supported by the German Federal Ministry of Education and Research under the project number 03WKCJ4D and the Eurostars projects DIESEL (E!9367) and QAMEL (E!9725) as well as the European Union’s H2020 research and innovation action HOBBIT under the Grant Agreement number 688227.

Publisher Copyright:
© 2018 IOS Press. All rights reserved.

DOI

Zuletzt angesehen

Publikationen

  1. Accuracy Improvement of Vision System for Mobile Robot Navigation by Finding the Energetic Center of Laser Signal
  2. Comparison of three methods of length compensation in a parallel kinematic and their equivalence conditions
  3. Graph-Based Early-Fusion for Flood Detection
  4. Deconstructing and reconstructing diversity in client-provider-relationships of social work
  5. Vielfalt des Alterns - Differenz oder Integration?
  6. ENVISIONING PROTECTED AREAS THROUGH PARTICIPATORY SCENARIO PLANNING: NAVIGATING COVERAGE AND EFFECTIVENESS CHALLENGES AHEAD
  7. A Control of an Electromagnetic Actuator Using Model Predictive Control
  8. Investigating quality raters' performance using interface evaluation methods
  9. Leaf trait variation within individuals mediates the relationship between tree species richness and productivity
  10. Effect of silicon content on hot working, processing maps, and microstructural evolution of cast TX32-0.4Al magnesium alloy
  11. Vimentin promoter methylation analysis is a suitable complement of a gene mutation marker panel for the detection of preneoplastic and neoplastic colonic lesions
  12. Searching for New Languages, Searching for Minor Voices in the Archive
  13. Internal reference price response across store formats
  14. Aspect-oriented software development
  15. Belowground top-down and aboveground bottom-up effects structure multitrophic community relationships in a biodiverse forest
  16. Sensorless Control of AC Motor Drives with Adaptive Extended Kalman Filter
  17. Self-supervised Siamese Autoencoders
  18. Towards greener and sustainable ionic liquids using naturally occurring and nature-inspired pyridinium structures
  19. Time use and time budgets
  20. Intelligence assessment with computer simulations
  21. Using photography to elicit grazier values and management practices relating to tree survival and recruitment
  22. Controlling a Bank Model Economy by Sliding Mode Control with Help of Kalman Filter
  23. Development and prospects of degradable magnesium alloys for structural and functional applications in the fields of environment and energy
  24. Conception and analysis of Cascaded Dual Kalman Filters as virtual sensors for mastication activity of stomatognathic craniomandibular system
  25. Integrating teacher and student workspaces in a technology-enhanced mathematics lecture
  26. Theme zones in English media discourse
  27. Simon Denny
  28. A Configurational Approach to Investigating the Relationship Between Organizational Culture and Organizational Effectiveness Using Fuzzy-Set Analysis
  29. In situ synchrotron diffraction of the solidification of Mg-RE alloys
  30. Placing Brazil's grasslands and savannas on the map of science and conservation
  31. Governance im Wandel
  32. Entrepreneurial actions
  33. IFIP WG 13.5 workshop on resilience, reliability, safety and human error in system development
  34. From niche to mainstream
  35. Sustainable Development Goals als Rahmenbedingung einer transformativen Berufsbildung
  36. A hybrid hydraulic piezo actuator modeling and hysteresis effect identification for control in camless internal combustion engines