QUANT - Question Answering Benchmark Curator

Publikation: Beiträge in SammelwerkenAufsätze in KonferenzbändenForschungbegutachtet

Standard

QUANT - Question Answering Benchmark Curator. / Gusmita, Ria Hari; Jalota, Rricha; Vollmers, Daniel et al.
Semantic Systems. The Power of AI and Knowledge Graphs - 15th International Conference, SEMANTiCS 2019 Proceedings. Hrsg. / Maribel Acosta; York Sure-Vetter; Philippe Cudré-Mauroux; Maria Maleshkova; Tassilo Pellegrini; Harald Sack. Springer, 2019. S. 343-358 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Band 11702 LNCS).

Publikation: Beiträge in SammelwerkenAufsätze in KonferenzbändenForschungbegutachtet

Harvard

Gusmita, RH, Jalota, R, Vollmers, D, Reineke, J, Ngonga Ngomo, AC & Usbeck, R 2019, QUANT - Question Answering Benchmark Curator. in M Acosta, Y Sure-Vetter, P Cudré-Mauroux, M Maleshkova, T Pellegrini & H Sack (Hrsg.), Semantic Systems. The Power of AI and Knowledge Graphs - 15th International Conference, SEMANTiCS 2019 Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Bd. 11702 LNCS, Springer, S. 343-358, 15th International Conference on Semantic Systems, SEMANTiCS 2019, Karlsruhe, Deutschland, 09.09.19. https://doi.org/10.1007/978-3-030-33220-4_25

APA

Gusmita, R. H., Jalota, R., Vollmers, D., Reineke, J., Ngonga Ngomo, A. C., & Usbeck, R. (2019). QUANT - Question Answering Benchmark Curator. In M. Acosta, Y. Sure-Vetter, P. Cudré-Mauroux, M. Maleshkova, T. Pellegrini, & H. Sack (Hrsg.), Semantic Systems. The Power of AI and Knowledge Graphs - 15th International Conference, SEMANTiCS 2019 Proceedings (S. 343-358). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Band 11702 LNCS). Springer. https://doi.org/10.1007/978-3-030-33220-4_25

Vancouver

Gusmita RH, Jalota R, Vollmers D, Reineke J, Ngonga Ngomo AC, Usbeck R. QUANT - Question Answering Benchmark Curator. in Acosta M, Sure-Vetter Y, Cudré-Mauroux P, Maleshkova M, Pellegrini T, Sack H, Hrsg., Semantic Systems. The Power of AI and Knowledge Graphs - 15th International Conference, SEMANTiCS 2019 Proceedings. Springer. 2019. S. 343-358. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-33220-4_25

Bibtex

@inbook{31b33b4964fc4fc4a7582f0970a1217d,
title = "QUANT - Question Answering Benchmark Curator",
abstract = "Question answering engines have become one of the most popular type of applications driven by Semantic Web technologies. Consequently, the provision of means to quantify the performance of current question answering approaches on current datasets has become ever more important. However, a large percentage of the queries found in popular question answering benchmarks cannot be executed on current versions of their reference dataset. There is a consequently a clear need to curate question answering benchmarks periodically. However, the manual alteration of question answering benchmarks is often error-prone. We alleviate this problem by presenting QUANT, a novel framework for the creation and curation of question answering benchmarks. QUANT supports the curation of benchmarks by generating smart edit suggestions for question-query pair and for the corresponding metadata. In addition, our framework supports the creation of new benchmark entries by providing predefined quality checks for queries. We evaluate QUANT on 653 questions obtained from QALD-1 to QALD-8 with 10 users. Our results show that our framework generates reliable suggestions and can reduce the curation effort for QA benchmarks by up to 91%.",
keywords = "Benchmark, Knowledge base, Question answering, Informatics, Business informatics",
author = "Gusmita, {Ria Hari} and Rricha Jalota and Daniel Vollmers and Jan Reineke and {Ngonga Ngomo}, {Axel Cyrille} and Ricardo Usbeck",
note = "This work was supported by the German Federal Ministry of Transport and Digital Infrastructure (BMVI) in the project LIMBO (no. 19F2029I) and by the German Federal Ministry of Education and Research (BMBF) in the project SOLIDE (no. 13N14456) within {\textquoteleft}KMU-innovativ: Forschung f{\"u}r die zivile Sicherheit{\textquoteright} in particular {\textquoteleft}Forschung f{\"u}r die zivile Sicherheit{\textquoteright}. Publisher Copyright: {\textcopyright} 2019, The Author(s).; 15th International Conference on Semantic Systems, SEMANTiCS 2019, SEMANTiCS 2019 ; Conference date: 09-09-2019 Through 12-09-2019",
year = "2019",
month = jan,
day = "1",
doi = "10.1007/978-3-030-33220-4_25",
language = "English",
isbn = "978-3-030-33219-8",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer",
pages = "343--358",
editor = "Maribel Acosta and York Sure-Vetter and Philippe Cudr{\'e}-Mauroux and Maria Maleshkova and Tassilo Pellegrini and Harald Sack",
booktitle = "Semantic Systems. The Power of AI and Knowledge Graphs - 15th International Conference, SEMANTiCS 2019 Proceedings",
address = "Germany",
url = "https://2019.semantics.cc/",

}

RIS

TY - CHAP

T1 - QUANT - Question Answering Benchmark Curator

AU - Gusmita, Ria Hari

AU - Jalota, Rricha

AU - Vollmers, Daniel

AU - Reineke, Jan

AU - Ngonga Ngomo, Axel Cyrille

AU - Usbeck, Ricardo

N1 - Conference code: 15

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Question answering engines have become one of the most popular type of applications driven by Semantic Web technologies. Consequently, the provision of means to quantify the performance of current question answering approaches on current datasets has become ever more important. However, a large percentage of the queries found in popular question answering benchmarks cannot be executed on current versions of their reference dataset. There is a consequently a clear need to curate question answering benchmarks periodically. However, the manual alteration of question answering benchmarks is often error-prone. We alleviate this problem by presenting QUANT, a novel framework for the creation and curation of question answering benchmarks. QUANT supports the curation of benchmarks by generating smart edit suggestions for question-query pair and for the corresponding metadata. In addition, our framework supports the creation of new benchmark entries by providing predefined quality checks for queries. We evaluate QUANT on 653 questions obtained from QALD-1 to QALD-8 with 10 users. Our results show that our framework generates reliable suggestions and can reduce the curation effort for QA benchmarks by up to 91%.

AB - Question answering engines have become one of the most popular type of applications driven by Semantic Web technologies. Consequently, the provision of means to quantify the performance of current question answering approaches on current datasets has become ever more important. However, a large percentage of the queries found in popular question answering benchmarks cannot be executed on current versions of their reference dataset. There is a consequently a clear need to curate question answering benchmarks periodically. However, the manual alteration of question answering benchmarks is often error-prone. We alleviate this problem by presenting QUANT, a novel framework for the creation and curation of question answering benchmarks. QUANT supports the curation of benchmarks by generating smart edit suggestions for question-query pair and for the corresponding metadata. In addition, our framework supports the creation of new benchmark entries by providing predefined quality checks for queries. We evaluate QUANT on 653 questions obtained from QALD-1 to QALD-8 with 10 users. Our results show that our framework generates reliable suggestions and can reduce the curation effort for QA benchmarks by up to 91%.

KW - Benchmark

KW - Knowledge base

KW - Question answering

KW - Informatics

KW - Business informatics

UR - http://www.scopus.com/inward/record.url?scp=85076221758&partnerID=8YFLogxK

UR - https://www.mendeley.com/catalogue/bd2d8832-d7a4-3b1e-99bc-89ac85f1e4bf/

U2 - 10.1007/978-3-030-33220-4_25

DO - 10.1007/978-3-030-33220-4_25

M3 - Article in conference proceedings

AN - SCOPUS:85076221758

SN - 978-3-030-33219-8

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 343

EP - 358

BT - Semantic Systems. The Power of AI and Knowledge Graphs - 15th International Conference, SEMANTiCS 2019 Proceedings

A2 - Acosta, Maribel

A2 - Sure-Vetter, York

A2 - Cudré-Mauroux, Philippe

A2 - Maleshkova, Maria

A2 - Pellegrini, Tassilo

A2 - Sack, Harald

PB - Springer

T2 - 15th International Conference on Semantic Systems, SEMANTiCS 2019

Y2 - 9 September 2019 through 12 September 2019

ER -

DOI

Zuletzt angesehen

Publikationen

  1. Policy schemes, operational strategies and system integration of residential co-generation fuel cells.
  2. Is there an excess of significant findings in published studies of psychotherapy for depression?
  3. Organic farming affects the biological control of hemipteran pests and yields in spring barley independent of landscape complexity
  4. Assessing Trust by Disclosure in Online Social Networks
  5. Product diversification and stability of employment and sales
  6. Dynamische Modellierung der Sorption von Substanzen in einem hydrologischen Einzugsgebietsmodell anhand des Beispiels Phosphor
  7. Macroeconomic shocks and banks’ foreign assets
  8. Improving compliance with COVID-19 guidance
  9. Predicting expatriate job performance
  10. Alcohol intake can reduce gambling behavior
  11. Statistical Learning and Inference Is Impaired in the Nonclinical Continuum of Psychosis
  12. Schulbezogenes mathematisches Vorwissen von Bewerber*innen auf ein Mathematik-Lehramtsstudium
  13. Plastics in our ocean as transdisciplinary challenge
  14. Investigation On The Influence Of Remanufacturing On Production Planning And Control – A Systematic Literature Review
  15. Stock price reactions to climate science information from the Intergovernmental Panel on Climate Change
  16. Effects of elevated atmospheric CO2 concentrations on barley, sugar beet and wheat in a rotation
  17. Toward “hardened” accountability?
  18. New evidence for vegetation development and timing of Upper Middle Pleistocene interglacials in Northern Germany and tentative correlations
  19. Intermetallic phase characteristics in the Mg–Nd–Zn system
  20. Does Internet-based guided self-help for depression cause harm?
  21. Print exposure across the reading life span
  22. Reaching for the (Product) Stars
  23. Influence of cerium on stress corrosion cracking in AZ91D
  24. Elution of Monomers from Two Conventional Dental Composite Materials
  25. Effects of introspective vs. extraspective instruction in scaling of hedonic properties of flavouring ingredients by Chinese and German subjects
  26. Simulation of the quench sensitivity of the aluminum alloy 6082
  27. Generalising IRT to Discriminate Between Examinees
  28. Systematic study of the effect of non-uniform seal stiffness on the contact stress in flat-faced soft-seated spring operated pressure relief valves
  29. Accurate welding line prediction in extrusion processes
  30. Photochemistry of 2,2', 4,4', %,5'-hexabde (BDE-153) in THF and adsorbed on SiO2
  31. Cues from Facial Expressions for Emotional Interfaces
  32. Manufacture of profiles with variable cross-sections, curvatures and twistings by influencing the material flow in hot extrusion
  33. Effects of habitat heterogeneity on bird communities in forests of northeastern Germany
  34. The multipole resonance probe
  35. German multiple-product, multiple-destination exporters: Bernard-Redding-Schott under test
  36. Forestry contributed to warming of forest ecosystems in northern Germany during the extreme summers of 2018 and 2019
  37. In Situ Synchrotron Radiation Study of the Tension–Compression Asymmetry in an Extruded Mg–2Y–1Zn–1Mn Alloy
  38. Entrepreneurial Traits and Strategy in the Performance of Owner-manager Led Firms
  39. Beyond Personalization and Anonymity:
  40. The effect of hunting regimes on tree regeneration in lowland beech (Fagus sylvatica L.) forests
  41. „Willst du wirklich ,lehren‘ oder ,Lehrer werden‘?“