GERBIL - General entity annotator benchmarking framework

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

  • Michael Röder
  • Axel Cyrille Ngonga Ngomo
  • Ciro Baron
  • Andreas Both
  • Martin Brümmer
  • Diego Ceccarelli
  • Marco Cornolti
  • Didier Cherix
  • Bernd Eickmann
  • Paolo Ferragina
  • Christiane Lemke
  • Andrea Moro
  • Roberto Navigli
  • Francesco Piccinno
  • Giuseppe Rizzo
  • Harald Sack
  • René Speck
  • Raphaël Troncy
  • Jörg Waitelonis
  • Lars Wesemann

The need to bridge between the unstructured data on the Document Web and the structured data on the Web of Data has led to the development of a considerable number of annotation tools. However, these tools are currently still hard to compare since the published evaluation results are calculated on diverse datasets and evaluated based on different measures. We present GERBIL, an evaluation framework for semantic entity annotation. The rationale behind our framework is to provide developers, end users and researchers with easy-To-use interfaces that allow for the agile, fine-grained and uniform evaluation of annotation tools on multiple datasets. By these means, we aim to ensure that both tool developers and end users can derive meaningful insights pertaining to the extension, integration and use of annotation applications. In particular, GERBIL provides comparable results to tool developers so as to allow them to easily discover the strengths and weaknesses of their implementations with respect to the state of the art. With the permanent experiment URIs provided by our framework, we ensure the reproducibility and archiving of evaluation results. Moreover, the framework generates data in machineprocessable format, allowing for the efficient querying and post-processing of evaluation results. Finally, the tool diagnostics provided by GERBIL allows deriving insights pertaining to the areas in which tools should be further refined, thus allowing developers to create an informed agenda for extensions and end users to detect the right tools for their purposes. GERBIL aims to become a focal point for the state of the art, driving the research agenda of the community by presenting comparable objective evaluation results.

Original languageEnglish
Title of host publicationWWW 2015 - Proceedings of the 24th International Conference on World Wide Web
EditorsAldo Gangemi, Stefano Leonardi, Alessandro Panconesi
Number of pages11
PublisherAssociation for Computing Machinery, Inc
Publication date18.05.2015
Pages1133-1143
ISBN (print)978-1-4503-3469-3
DOIs
Publication statusPublished - 18.05.2015
Externally publishedYes
Event24th International Conference on World Wide Web, WWW 2015 - Florence, Italy
Duration: 18.05.201522.05.2015
https://dl.acm.org/doi/proceedings/10.1145/2740908
https://dblp.org/db/conf/www/www2015.html

Links

DOI

Recently viewed

Researchers

  1. Marcus Erben

Activities

  1. Unboxing Uncertainties – Interrogating Forecasting and Foreclosing Future Infrastructures in the Amazon Town
  2. Efficacy of an app-based gratitude intervention in reducing repetitive negative thinking and fostering resilience: results of a randomized controlled trial
  3. "Curious and Concerned" – A mixed-methods study of teacher educators’ AI literacy, usage experience, and perceptions
  4. Es geht um
  5. The Relation of Children's Performances in Spatial Tasks at Two Different Scales of Space
  6. An Axiomatic Approach to Decision under Knightian Uncertainty
  7. Plasma shock wave simulation for laser shock processing
  8. Ars Electronica
  9. Workshop Medzin I
  10. Performance resource depletion influence on performance: Advancing concepts and findings
  11. Assessing cognitive abilities for L2 learning: A review of accessible tools
  12. Simulation and Evaluation of Control Mechanisms for Mobile Robot Fulfillment Systems
  13. Towards an Undercommons (Eco)Logistics?
  14. Knowledge Mobilization in Open Innovation Networks: What’s in It for Schools?
  15. Workshop on "The State and beyond: Actor constellations in resource conflicts" - 2015
  16. Artificial Intelligence and Intellectual Property
  17. Is a better understanding of assembly a way to help reassemble communities for restoration?
  18. The golden age of software architecture better named the middle age of software architecture - Some provocative thoughts
  19. EXIT
  20. From Magic to Systemics. Heinz von Foerster and the Reenchantment of Science
  21. Co-supervisor of the dissertation "Diversity and functions of plant-insect interactions along a forest retention gradient"
  22. ICSB World Conference - ICSB 2007

Publications

  1. Rapid grain refinement and compositional homogenization in a cast binary Cu50Ni alloy achieved by friction stir processing
  2. Mathematical relation between extended connectivity and eigenvector coefficients.
  3. Integrating the underlying structure of stochasticity into community ecology
  4. Towards productive functions?
  5. Inside-sediment partitioning of PAH, PCB and organochlorine compounds and inferences on sampling and normalization methods
  6. The Structure of Student Interest in Computers and Information Technology
  7. Monitoring of microbially mediated corrosion and scaling processes using redox potential measurements
  8. Managing complexity in automative production
  9. Scholarly Question Answering Using Large Language Models in the NFDI4DataScience Gateway
  10. A Computational Research System for the History of Science
  11. What Makes for a Good Theory? How to Evaluate a Theory Using the Strength Model of Self-Control as an Example
  12. "And I Think That Is a Very Straightforward Way of Dealing With It''
  13. Tuning kalman filter in linear systems
  14. Measuring cognitive load with subjective rating scales during problem solving
  15. Optimal dynamic scale and structure of a multi-pollution economy
  16. Applications of the Simultaneous Modular Approach in the Field of Material Flow Analysis
  17. Scale-dependent diversity patterns affect spider assemblages of two contrasting forest ecosystems
  18. 8th challenge on question answering over linked data (QALD-8)
  19. Operations Systems of Container Terminals
  20. Entity linking in 40 languages using MAG
  21. Object-Oriented Construction Handbook
  22. Recognition and approach responses toward threatening objects