QUANT - Question Answering Benchmark Curator

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

  • Ria Hari Gusmita
  • Rricha Jalota
  • Daniel Vollmers
  • Jan Reineke
  • Axel Cyrille Ngonga Ngomo
  • Ricardo Usbeck

Question answering engines have become one of the most popular type of applications driven by Semantic Web technologies. Consequently, the provision of means to quantify the performance of current question answering approaches on current datasets has become ever more important. However, a large percentage of the queries found in popular question answering benchmarks cannot be executed on current versions of their reference dataset. There is a consequently a clear need to curate question answering benchmarks periodically. However, the manual alteration of question answering benchmarks is often error-prone. We alleviate this problem by presenting QUANT, a novel framework for the creation and curation of question answering benchmarks. QUANT supports the curation of benchmarks by generating smart edit suggestions for question-query pair and for the corresponding metadata. In addition, our framework supports the creation of new benchmark entries by providing predefined quality checks for queries. We evaluate QUANT on 653 questions obtained from QALD-1 to QALD-8 with 10 users. Our results show that our framework generates reliable suggestions and can reduce the curation effort for QA benchmarks by up to 91%.

Original languageEnglish
Title of host publicationSemantic Systems. The Power of AI and Knowledge Graphs - 15th International Conference, SEMANTiCS 2019 Proceedings
EditorsMaribel Acosta, York Sure-Vetter, Philippe Cudré-Mauroux, Maria Maleshkova, Tassilo Pellegrini, Harald Sack
Number of pages16
PublisherSpringer Verlag
Publication date01.01.2019
Pages343-358
ISBN (print)978-3-030-33219-8
ISBN (electronic)978-3-030-33220-4
DOIs
Publication statusPublished - 01.01.2019
Externally publishedYes
Event15th International Conference on Semantic Systems, SEMANTiCS 2019 - Karlsruhe, Germany
Duration: 09.09.201912.09.2019
Conference number: 15
https://2019.semantics.cc/

Bibliographical note

This work was supported by the German Federal Ministry of Transport and Digital Infrastructure (BMVI) in the project LIMBO (no. 19F2029I) and by the German Federal Ministry of Education and Research (BMBF) in the project SOLIDE (no. 13N14456) within ‘KMU-innovativ: Forschung für die zivile Sicherheit’ in particular ‘Forschung für die zivile Sicherheit’.

Publisher Copyright:
© 2019, The Author(s).

Recently viewed

Publications

  1. From Allegiant to Assertive Citizens
  2. Feature selection for density level-sets
  3. Assessing the Bonding Interface Characteristics and Mechanical Properties of Bobbin Tool Friction Stir Welded Dissimilar Aluminum Alloy Joints
  4. Aboveground overyielding in grassland mixtures is associated with reduced biomass partitioning to belowground organs
  5. Artificial neural network for correction of effects of plasticity in equibiaxial residual stress profiles measured by hole drilling
  6. Semi-polar root exudates in natural grassland communities
  7. Using rating scales for the assessment of physical self-concept
  8. Perceptions of Organizational Downsizing
  9. Model of mobility demands for future short distance public transport systems
  10. Process limits of extrusion of multimaterial components
  11. Steady State Detection for the Context Aware Evaluation of Vital Signs
  12. Where there is no World and no Epoch
  13. Introduction
  14. Problem Definition and Agenda-Setting in Critical Perspective
  15. Validation of an online imitation-inhibition task
  16. The 1986 Principles Relating to Remote Sensing of the Earth from Outer Space (RS Princi­ples)
  17. Rating Player Actions in Soccer
  18. Linking concepts of change and ecosystem services research: A systematic review
  19. Two model formulations for gradient crystal plasticity
  20. Material utilization of organic residues
  21. Augmented space: Real, virtual and symbolic spaces, a workshop held at the Universitat Luneburg, Germany, July 2000
  22. Feel the Music! Exploring the Cross-modal Correspondence between Music and Haptic Perceptions of Softness
  23. Can adults learn L2 grammar after prolonged exposure under incidental conditions?
  24. Understanding Societies from Inside the Organisms
  25. Expert*inneninterview
  26. Geodetic rays and fibers in periodic graphs
  27. Mechanisms of teleological change
  28. Efficacy of a Self-Help Web-Based Recovery Training in Improving Sleep in Workers
  29. Value of large-scale linear networks for bird conservation