QUANT - Question Answering Benchmark Curator

Publikation: Beiträge in SammelwerkenAufsätze in KonferenzbändenForschungbegutachtet

Authors

  • Ria Hari Gusmita
  • Rricha Jalota
  • Daniel Vollmers
  • Jan Reineke
  • Axel Cyrille Ngonga Ngomo
  • Ricardo Usbeck

Question answering engines have become one of the most popular type of applications driven by Semantic Web technologies. Consequently, the provision of means to quantify the performance of current question answering approaches on current datasets has become ever more important. However, a large percentage of the queries found in popular question answering benchmarks cannot be executed on current versions of their reference dataset. There is a consequently a clear need to curate question answering benchmarks periodically. However, the manual alteration of question answering benchmarks is often error-prone. We alleviate this problem by presenting QUANT, a novel framework for the creation and curation of question answering benchmarks. QUANT supports the curation of benchmarks by generating smart edit suggestions for question-query pair and for the corresponding metadata. In addition, our framework supports the creation of new benchmark entries by providing predefined quality checks for queries. We evaluate QUANT on 653 questions obtained from QALD-1 to QALD-8 with 10 users. Our results show that our framework generates reliable suggestions and can reduce the curation effort for QA benchmarks by up to 91%.

OriginalspracheEnglisch
TitelSemantic Systems. The Power of AI and Knowledge Graphs - 15th International Conference, SEMANTiCS 2019 Proceedings
HerausgeberMaribel Acosta, York Sure-Vetter, Philippe Cudré-Mauroux, Maria Maleshkova, Tassilo Pellegrini, Harald Sack
Anzahl der Seiten16
VerlagSpringer Verlag
Erscheinungsdatum01.01.2019
Seiten343-358
ISBN (Print)978-3-030-33219-8
ISBN (elektronisch)978-3-030-33220-4
DOIs
PublikationsstatusErschienen - 01.01.2019
Extern publiziertJa
Veranstaltung15th International Conference on Semantic Systems, SEMANTiCS 2019 - Karlsruhe, Deutschland
Dauer: 09.09.201912.09.2019
Konferenznummer: 15
https://2019.semantics.cc/

DOI

Zuletzt angesehen

Forschende

  1. Michael Besser

Publikationen

  1. Students' Time Allocation and School Performance
  2. Sustainable engineering education in research and practice
  3. How health message framing and targets affect distancing during the Covid-19 pandemic
  4. A Social–Ecological Systems Framework as a Tool for Understanding the Effectiveness of Biosphere Reserve Management
  5. Co-productive agility and four collaborative pathways to sustainability transformations
  6. Impacts of entrepreneur’s error orientation on performance: A cross-culture comparison
  7. Speaking about vision, talking in the name of so much more
  8. Synchronic and Diachronic Pragmatic Variability
  9. The self-sabotage of conservation
  10. Globalization and the societal consensus of wealth tax cuts
  11. Discrimination at work: Effects on Job Satisfaction and Organizational Commitment. An empirical study of the influence of perceived discrimination on work-related behaviours among people with and without a migration background
  12. Influence of Torsion on Precipitation and Hardening Effects during Aging of an Extruded AZ91 Alloy
  13. Vermittlungstheologie II. Dogmatisch
  14. Spielt Charlie Parker in den Wind oder mit ihm?
  15. From the environmental state to the sustainability state? Conceptualization, indicators, and examples
  16. Action and action-regulation in entrepreneurship: Evaluating a student training for promoting entrepreneurship
  17. Wege in eine bessere Zukunft der Hochschulen
  18. Threshold stress during tensile and compressive creep in AE42 magnesium alloy
  19. General conclusions
  20. Tuning into Things
  21. Facing complex crime
  22. Public service media, innovation policy and the ‘crowding out’ problem
  23. Does a smile open all doors? Understanding the impact of appearance disclosure on accommodation sharing platforms
  24. Space-focused stereotypes and their potential role in group-based disparities in social work services
  25. High-Volume Resistance Training Improves Double-Poling Peak Oxygen Uptake in Youth Elite Cross-Country Skiers and Biathletes
  26. A Performance Motivator in one Country, A Non-Motivator in Another?
  27. Logic as a Medium
  28. Experimental-numerical study of laser-shock-peening-induced retardation of fatigue crack propagation in Ti-17 titanium alloy
  29. Farewell to the party model?
  30. Zinc and cadmium accumulation in single zebrafish (Danio rerio) embryos - A total reflection X-ray fluorescence spectrometry application
  31. Kosmopolitismus
  32. Debattieren im Unterricht
  33. A review of ecosystem service benefits from wild bees across social contexts
  34. European hospitals as a Source for Platinum in the Environment in Comparison with Other Sources
  35. Reconciling Analytics with Holistic Thinking in Business Sustainability Decision-Making
  36. The priority value of scrubland habitats for carnivore conservation in Mediterranean ecosystems
  37. Modern Micropolitics of Antipopulism
  38. Datenbasierte Selbst- und Fremdevaluation
  39. Developments in Qualitative Mindfulness Practice Research
  40. "Das muss man umdrehen und dann passt es"
  41. Community-based Entrepreneurship and Rural Development
  42. Inmitten der Extreme