QUANT - Question Answering Benchmark Curator

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

  • Ria Hari Gusmita
  • Rricha Jalota
  • Daniel Vollmers
  • Jan Reineke
  • Axel Cyrille Ngonga Ngomo
  • Ricardo Usbeck

Question answering engines have become one of the most popular type of applications driven by Semantic Web technologies. Consequently, the provision of means to quantify the performance of current question answering approaches on current datasets has become ever more important. However, a large percentage of the queries found in popular question answering benchmarks cannot be executed on current versions of their reference dataset. There is a consequently a clear need to curate question answering benchmarks periodically. However, the manual alteration of question answering benchmarks is often error-prone. We alleviate this problem by presenting QUANT, a novel framework for the creation and curation of question answering benchmarks. QUANT supports the curation of benchmarks by generating smart edit suggestions for question-query pair and for the corresponding metadata. In addition, our framework supports the creation of new benchmark entries by providing predefined quality checks for queries. We evaluate QUANT on 653 questions obtained from QALD-1 to QALD-8 with 10 users. Our results show that our framework generates reliable suggestions and can reduce the curation effort for QA benchmarks by up to 91%.

Original languageEnglish
Title of host publicationSemantic Systems. The Power of AI and Knowledge Graphs - 15th International Conference, SEMANTiCS 2019 Proceedings
EditorsMaribel Acosta, York Sure-Vetter, Philippe Cudré-Mauroux, Maria Maleshkova, Tassilo Pellegrini, Harald Sack
Number of pages16
PublisherSpringer Verlag
Publication date01.01.2019
Pages343-358
ISBN (print)978-3-030-33219-8
ISBN (electronic)978-3-030-33220-4
DOIs
Publication statusPublished - 01.01.2019
Externally publishedYes
Event15th International Conference on Semantic Systems, SEMANTiCS 2019 - Karlsruhe, Germany
Duration: 09.09.201912.09.2019
Conference number: 15
https://2019.semantics.cc/

Bibliographical note

This work was supported by the German Federal Ministry of Transport and Digital Infrastructure (BMVI) in the project LIMBO (no. 19F2029I) and by the German Federal Ministry of Education and Research (BMBF) in the project SOLIDE (no. 13N14456) within ‘KMU-innovativ: Forschung für die zivile Sicherheit’ in particular ‘Forschung für die zivile Sicherheit’.

Publisher Copyright:
© 2019, The Author(s).

Recently viewed

Publications

  1. Conclusions
  2. An inquiry into the digitisation of border and migration management
  3. Terror Management Theory and the Existential Psychological Function of Heroes
  4. „More than a game“
  5. The effects of an Internet based self-help course for reducing panic symptoms-Don't Panic Online
  6. A black box identification in frequency domain
  7. Assessing the environmental fate of S-metolachlor, its commercial product Mercantor Gold® and their photoproducts using a water-sediment test and in silico methods
  8. Using machine learning to identify important predictors of COVID-19 infection prevention behaviors during the early phase of the pandemic
  9. Proof of Stake
  10. Infinite Mixtures of Markov Chains
  11. Inflation Narratives from a Machine Learning Perspective
  12. Fatigue crack propagation in AA5083 structures additively manufactured via multi-layer friction surfacing
  13. SMARTPHONE APPS FOR TINNITUS: A REVIEW ON INTERVENTION COMPONENTS AND BEHAVIOR CHANGE TECHNIQUES USED IN TINNITUS APPS
  14. Impacts of urban real-world labs: Insights from a co-evaluation process informed by structuration theory in Wuppertal-Mirke
  15. Ein un(mögliches) Programm
  16. Development of a procedure for forming assisted thermal joining of tubes
  17. Velocity planning to optimize traction losses in a City-Bus Equipped with Permanent Magnet Three-Phase Synchronous Motors
  18. European External Action Service
  19. Econometric Studies with Integrated Micro-data from German Official Statistics
  20. Influence of 8-weeks of supervised static stretching or resistance training of pectoral major muscles on maximal strength, muscle thickness and range of motion
  21. When the petting zoo spawns into monsters
  22. Eine Koalition auf Kosten des Klimas
  23. The informed society
  24. The effects of psychological treatment of maternal depression on children and parental functioning
  25. Machen Urlaubsreisen glücklich?
  26. Health promoting schools and COVID-19
  27. Differenzielles Lernen im Golf
  28. Reduced nitrate leaching from an Irish cropland soil under non-inversion tillage with cover cropping greatly outweighs increased dissolved organic nitrogen leaching
  29. On the evidence for human use and control of fire at Schöningen
  30. Democratising platform governance in the sharing economy
  31. Measuring Effective Democracy
  32. Research contributions to personality at work
  33. Subject Citizen. The Ambiguity of Citizenship and its Colonial Laboratories
  34. Environmental Value Added
  35. Nachhaltigkeit
  36. Assessing sustainable biophysical human-nature connectedness at regional scales
  37. Forschung zu Energiewende und Partizipation
  38. Metamorphosen des Alltags
  39. Ein unwetter in Jerusalem
  40. Einleitung
  41. Oscar Wilde, "Der glückliche Prinz und andere Märchen"
  42. Lehren und Lernen - aber wie?
  43. Das Subjekt und die Atmosphären, durch die es ist
  44. Antispecismi e letture di Marx
  45. Investigating the Persuasive Effects of Testimonials on the Acceptance of Digital Stress Management Trainings Among University Students and Underlying Mechanisms
  46. Wie entscheiden Initiatoren und Backer im Reward based Crowdfunding?
  47. Wie kann Exnovation politisch gestaltet werden?
  48. Theoretische Rahmungen von (Gewalt-) Situationen