GERBIL - General entity annotator benchmarking framework

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

  • Michael Röder
  • Axel Cyrille Ngonga Ngomo
  • Ciro Baron
  • Andreas Both
  • Martin Brümmer
  • Diego Ceccarelli
  • Marco Cornolti
  • Didier Cherix
  • Bernd Eickmann
  • Paolo Ferragina
  • Christiane Lemke
  • Andrea Moro
  • Roberto Navigli
  • Francesco Piccinno
  • Giuseppe Rizzo
  • Harald Sack
  • René Speck
  • Raphaël Troncy
  • Jörg Waitelonis
  • Lars Wesemann

The need to bridge between the unstructured data on the Document Web and the structured data on the Web of Data has led to the development of a considerable number of annotation tools. However, these tools are currently still hard to compare since the published evaluation results are calculated on diverse datasets and evaluated based on different measures. We present GERBIL, an evaluation framework for semantic entity annotation. The rationale behind our framework is to provide developers, end users and researchers with easy-To-use interfaces that allow for the agile, fine-grained and uniform evaluation of annotation tools on multiple datasets. By these means, we aim to ensure that both tool developers and end users can derive meaningful insights pertaining to the extension, integration and use of annotation applications. In particular, GERBIL provides comparable results to tool developers so as to allow them to easily discover the strengths and weaknesses of their implementations with respect to the state of the art. With the permanent experiment URIs provided by our framework, we ensure the reproducibility and archiving of evaluation results. Moreover, the framework generates data in machineprocessable format, allowing for the efficient querying and post-processing of evaluation results. Finally, the tool diagnostics provided by GERBIL allows deriving insights pertaining to the areas in which tools should be further refined, thus allowing developers to create an informed agenda for extensions and end users to detect the right tools for their purposes. GERBIL aims to become a focal point for the state of the art, driving the research agenda of the community by presenting comparable objective evaluation results.

Original languageEnglish
Title of host publicationWWW 2015 - Proceedings of the 24th International Conference on World Wide Web
EditorsAldo Gangemi, Stefano Leonardi, Alessandro Panconesi
Number of pages11
PublisherAssociation for Computing Machinery, Inc
Publication date18.05.2015
Pages1133-1143
ISBN (print)978-1-4503-3469-3
DOIs
Publication statusPublished - 18.05.2015
Externally publishedYes
Event24th International Conference on World Wide Web, WWW 2015 - Florence, Italy
Duration: 18.05.201522.05.2015
https://dl.acm.org/doi/proceedings/10.1145/2740908
https://dblp.org/db/conf/www/www2015.html

Links

DOI

Recently viewed

Publications

  1. Spaces for challenging experiences, indeterminacy, and experimentation
  2. Anwendungsprogrammierung mit Embedded-SQL
  3. Portuguese part-of-speech tagging with large margin structure learning
  4. Geometric structures for the parameterization of non-interacting dynamics for multi-body mechanisms
  5. Learning linear classifiers sensitive to example dependent and noisy costs
  6. Simple measures and complex structures
  7. Quantification and analysis of surface macroplastic contamination on arable areas
  8. Emergence of Responsiveness Across Organizations, Networks, and Clusters from a Dynamic Capability Perspective
  9. State-wide university implementation of an online platform for eating disorders screening and intervention.
  10. Studying embodied encounters
  11. Set oriented computation of transport rates in 3-degree of freedom systems
  12. Generative 3D reconstruction of Ti-6Al-4V basketweave microstructures by optimization of differentiable microstructural descriptors
  13. Manufacturing, control, and performance evaluation of a Gecko-inspired soft robot
  14. Embodiment of Science in Science Slams.
  15. More than a YouTube Channel
  16. Generalizing Trust
  17. Attention and the Speed of Information Processing
  18. Building capacity for the science-policy interface on biodiversity and ecosystem services
  19. Export entry, export exit, and productivity in German manufacturing industries
  20. Managing Global Production Networks
  21. A Bayesian EAP-Based Nonlinear Extension of Croon and Van Veldhoven’s Model for Analyzing Data from Micro–Macro Multilevel Designs
  22. Allometric equations for maximum filtration rate in blue mussels Mytilus edulis and importance of condition index
  23. Gross, Richard. Understanding Grief: An Introduction, Routledge, 2016
  24. Cross-Fertilizing Qualitative Perspectives on Effects of a Mindfulness-Based Intervention: An Empirical Comparison of Four Methodical Approaches
  25. How attribution-of-competence and scale-granularity explain the anchor precision effect in negotiations and estimations.
  26. Keep calm and follow the news
  27. Utopian Hacks
  28. Developing European conservation and mitigation tools for pollination services: approaches of the STEP (Status and Trends of European Pollinators) project
  29. History of Embryology: Visualizations Through Series and Animation
  30. Modeling and predicting aquatic aerobic biodegradation