Treating dialogue quality evaluation as an anomaly detection problem

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

Dialogue systems for interaction with humans have been enjoying increased popularity in the research and industry fields. To this day, the best way to estimate their success is through means of human evaluation and not automated approaches, despite the abundance of work done in the field. In this paper, we investigate the effectiveness of perceiving dialogue evaluation as an anomaly detection task. The paper looks into four dialogue modeling approaches and how their objective functions correlate with human annotation scores. A high-level perspective exhibits negative results. However, a more in-depth look shows limited potential for using anomaly detection for evaluating dialogues.

Original languageEnglish
Title of host publicationLREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings
EditorsNicoletta Calzolari, Frederic Bechet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Number of pages5
PublisherEuropean Language Resources Association (ELRA)
Publication date2020
Pages508-512
ISBN (electronic)9791095546344
Publication statusPublished - 2020
Externally publishedYes
Event12th International Conference on Language Resources and Evaluation, LREC 2020 - Le Palais du Pharao, Marseille, France
Duration: 11.05.202016.05.2020
https://lrec2020.lrec-conf.org/en/about/organizers/index.html

Bibliographical note

Publisher Copyright:
© European Language Resources Association (ELRA), licensed under CC-BY-NC

Links

Recently viewed

Publications

  1. Saving (in) a common world
  2. Robust Control of Excavation Mobile Robot with Dynamic Triangulation Vision
  3. Other spaces
  4. Moving Towards Measuring Multifunctionality in Ecosystems: FieldScreen – A Mobile Positioning System for Non-Invasive Measurement of Plant Traits in Field Experiments
  5. Schooling, local knowledge and working memory
  6. archiDART: an R package for the automated computation of plant root architectural traits
  7. § 37a
  8. Traffic Life: Temporal Dynamics and Regulatory Dimensions in Agent-Based Transport Simulations
  9. Temporal Dynamics of Ecosystem Services
  10. QALD-10 — The 10th Challenge on Question Answering over Linked Data
  11. An Adaptive Resonance Regulator for an Actuator using Periodic Signals in Camless Engine Systems
  12. Class size, student performance and Tiebout bias
  13. The role of plant biodiversity in modifying the structure and functioning of higher tropic Levels in species-rich forests
  14. Explaining implementation deficits through multi-level governance in the EU's new member states
  15. Non-destructive transmissive inductive thickness sensor for IoT applications
  16. Tree diversity promotes functional dissimilarity and maintains functional richness despite species loss in predator assemblages
  17. Program for Better Riding
  18. Erratum: Formalised and non-formalised methods in resource management-knowledge and social learning in participatory processes
  19. Nonlinear control allocation applied on a QTR
  20. Socio-technical instruments in the field of Integrated Water Resources Management
  21. Nitrogen uptake by grassland communities
  22. Deep drawing of high-strength tailored blanks by using tailored tools
  23. Dynamic Semantic Web Content for Museum Guides
  24. Identification of Parameters and States in PMSMs
  25. Erroneous Examples: A Preliminary Investigation into Learning Benefits
  26. Reference wages and turnover intentions
  27. For whom are internet-based occupational mental health interventions effective? Moderators of internet-based problem-solving training outcome