Treating dialogue quality evaluation as an anomaly detection problem

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

Dialogue systems for interaction with humans have been enjoying increased popularity in the research and industry fields. To this day, the best way to estimate their success is through means of human evaluation and not automated approaches, despite the abundance of work done in the field. In this paper, we investigate the effectiveness of perceiving dialogue evaluation as an anomaly detection task. The paper looks into four dialogue modeling approaches and how their objective functions correlate with human annotation scores. A high-level perspective exhibits negative results. However, a more in-depth look shows limited potential for using anomaly detection for evaluating dialogues.

Original languageEnglish
Title of host publicationLREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings
EditorsNicoletta Calzolari, Frederic Bechet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Number of pages5
PublisherEuropean Language Resources Association (ELRA)
Publication date2020
Pages508-512
ISBN (electronic)9791095546344
Publication statusPublished - 2020
Externally publishedYes
Event12th International Conference on Language Resources and Evaluation, LREC 2020 - Le Palais du Pharao, Marseille, France
Duration: 11.05.202016.05.2020
https://lrec2020.lrec-conf.org/en/about/organizers/index.html

Bibliographical note

Publisher Copyright:
© European Language Resources Association (ELRA), licensed under CC-BY-NC

Links

Recently viewed

Publications

  1. Offline question answering over linked data using limited resources
  2. Evaluating structural and compositional canopy characteristics to predict the light-demand signature of the forest understorey in mixed, semi-natural temperate forests
  3. Primary Side Circuit Design of a Multi-coil Inductive System for Powering Wireless Sensors
  4. Support vector machines with example dependent costs
  5. A Playful Approach to Interactive Media in the Foreign Language Classroom
  6. Complex problem solving and intelligence
  7. Metaheuristics approach for solving personalized crew rostering problem in public bus transit
  8. Experiments on the Fehrer-Raab effect and the ‘Weather Station Model’ of visual backward masking
  9. Taking the pulse of Earth's tropical forests using networks of highly distributed plots
  10. Automatic three-dimensional geometry and mesh generation of periodic representative volume elements for matrix-inclusion composites
  11. Introduction: The representative turn in EU studies
  12. GERBIL - General entity annotator benchmarking framework
  13. Rapid grain refinement and compositional homogenization in a cast binary Cu50Ni alloy achieved by friction stir processing
  14. Mathematical relation between extended connectivity and eigenvector coefficients.
  15. Integrating the underlying structure of stochasticity into community ecology
  16. Towards productive functions?
  17. Inside-sediment partitioning of PAH, PCB and organochlorine compounds and inferences on sampling and normalization methods
  18. The Structure of Student Interest in Computers and Information Technology
  19. Monitoring of microbially mediated corrosion and scaling processes using redox potential measurements
  20. Managing complexity in automative production
  21. Scholarly Question Answering Using Large Language Models in the NFDI4DataScience Gateway
  22. A Computational Research System for the History of Science
  23. What Makes for a Good Theory? How to Evaluate a Theory Using the Strength Model of Self-Control as an Example
  24. "And I Think That Is a Very Straightforward Way of Dealing With It''