Treating dialogue quality evaluation as an anomaly detection problem

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Standard

Treating dialogue quality evaluation as an anomaly detection problem. / Nedelchev, Rostislav; Lehmann, Jens; Usbeck, Ricardo.
LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings. ed. / Nicoletta Calzolari; Frederic Bechet; Philippe Blache; Khalid Choukri; Christopher Cieri; Thierry Declerck; Sara Goggi; Hitoshi Isahara; Bente Maegaard; Joseph Mariani; Helene Mazo; Asuncion Moreno; Jan Odijk; Stelios Piperidis. European Language Resources Association (ELRA), 2020. p. 508-512 (LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings).

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Harvard

Nedelchev, R, Lehmann, J & Usbeck, R 2020, Treating dialogue quality evaluation as an anomaly detection problem. in N Calzolari, F Bechet, P Blache, K Choukri, C Cieri, T Declerck, S Goggi, H Isahara, B Maegaard, J Mariani, H Mazo, A Moreno, J Odijk & S Piperidis (eds), LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings. LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings, European Language Resources Association (ELRA), pp. 508-512, 12th International Conference on Language Resources and Evaluation, LREC 2020, Marseille, France, 11.05.20. <https://aclanthology.org/2020.lrec-1.64>

APA

Nedelchev, R., Lehmann, J., & Usbeck, R. (2020). Treating dialogue quality evaluation as an anomaly detection problem. In N. Calzolari, F. Bechet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, & S. Piperidis (Eds.), LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings (pp. 508-512). (LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings). European Language Resources Association (ELRA). https://aclanthology.org/2020.lrec-1.64

Vancouver

Nedelchev R, Lehmann J, Usbeck R. Treating dialogue quality evaluation as an anomaly detection problem. In Calzolari N, Bechet F, Blache P, Choukri K, Cieri C, Declerck T, Goggi S, Isahara H, Maegaard B, Mariani J, Mazo H, Moreno A, Odijk J, Piperidis S, editors, LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings. European Language Resources Association (ELRA). 2020. p. 508-512. (LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings).

Bibtex

@inbook{4e2b83e41f414cb48aa475435e6918da,
title = "Treating dialogue quality evaluation as an anomaly detection problem",
abstract = "Dialogue systems for interaction with humans have been enjoying increased popularity in the research and industry fields. To this day, the best way to estimate their success is through means of human evaluation and not automated approaches, despite the abundance of work done in the field. In this paper, we investigate the effectiveness of perceiving dialogue evaluation as an anomaly detection task. The paper looks into four dialogue modeling approaches and how their objective functions correlate with human annotation scores. A high-level perspective exhibits negative results. However, a more in-depth look shows limited potential for using anomaly detection for evaluating dialogues.",
keywords = "Dialogue, Discourse Annotation, Evaluation Methodologies, Processing, Representation, Informatics, Business informatics",
author = "Rostislav Nedelchev and Jens Lehmann and Ricardo Usbeck",
note = "Publisher Copyright: {\textcopyright} European Language Resources Association (ELRA), licensed under CC-BY-NC; 12th International Conference on Language Resources and Evaluation, LREC 2020 ; Conference date: 11-05-2020 Through 16-05-2020",
year = "2020",
language = "English",
series = "LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings",
publisher = "European Language Resources Association (ELRA)",
pages = "508--512",
editor = "Nicoletta Calzolari and Frederic Bechet and Philippe Blache and Khalid Choukri and Christopher Cieri and Thierry Declerck and Sara Goggi and Hitoshi Isahara and Bente Maegaard and Joseph Mariani and Helene Mazo and Asuncion Moreno and Jan Odijk and Stelios Piperidis",
booktitle = "LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings",
address = "Luxembourg",
url = "https://lrec2020.lrec-conf.org/en/about/organizers/index.html",

}

RIS

TY - CHAP

T1 - Treating dialogue quality evaluation as an anomaly detection problem

AU - Nedelchev, Rostislav

AU - Lehmann, Jens

AU - Usbeck, Ricardo

N1 - Publisher Copyright: © European Language Resources Association (ELRA), licensed under CC-BY-NC

PY - 2020

Y1 - 2020

N2 - Dialogue systems for interaction with humans have been enjoying increased popularity in the research and industry fields. To this day, the best way to estimate their success is through means of human evaluation and not automated approaches, despite the abundance of work done in the field. In this paper, we investigate the effectiveness of perceiving dialogue evaluation as an anomaly detection task. The paper looks into four dialogue modeling approaches and how their objective functions correlate with human annotation scores. A high-level perspective exhibits negative results. However, a more in-depth look shows limited potential for using anomaly detection for evaluating dialogues.

AB - Dialogue systems for interaction with humans have been enjoying increased popularity in the research and industry fields. To this day, the best way to estimate their success is through means of human evaluation and not automated approaches, despite the abundance of work done in the field. In this paper, we investigate the effectiveness of perceiving dialogue evaluation as an anomaly detection task. The paper looks into four dialogue modeling approaches and how their objective functions correlate with human annotation scores. A high-level perspective exhibits negative results. However, a more in-depth look shows limited potential for using anomaly detection for evaluating dialogues.

KW - Dialogue

KW - Discourse Annotation

KW - Evaluation Methodologies

KW - Processing

KW - Representation

KW - Informatics

KW - Business informatics

UR - http://www.scopus.com/inward/record.url?scp=85096532825&partnerID=8YFLogxK

UR - https://www.mendeley.com/catalogue/50653d2d-f7e2-36ae-871a-85fb7753b95b/

M3 - Article in conference proceedings

AN - SCOPUS:85096532825

T3 - LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings

SP - 508

EP - 512

BT - LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings

A2 - Calzolari, Nicoletta

A2 - Bechet, Frederic

A2 - Blache, Philippe

A2 - Choukri, Khalid

A2 - Cieri, Christopher

A2 - Declerck, Thierry

A2 - Goggi, Sara

A2 - Isahara, Hitoshi

A2 - Maegaard, Bente

A2 - Mariani, Joseph

A2 - Mazo, Helene

A2 - Moreno, Asuncion

A2 - Odijk, Jan

A2 - Piperidis, Stelios

PB - European Language Resources Association (ELRA)

T2 - 12th International Conference on Language Resources and Evaluation, LREC 2020

Y2 - 11 May 2020 through 16 May 2020

ER -

Links

Recently viewed

Publications

  1. Non-destructive transmissive inductive thickness sensor for IoT applications
  2. Spielt es nur eine Rolle "was" gepromptet wird oder auch "wann" gepromptet wird.
  3. Mindfulness and cognitive-behavioral strategies for psychological detachment
  4. Multiscale analysis for the bio-heat transfer equation - The nonisolated case
  5. Ein piezohydraulischer vollvariabler Ventilantrieb eines Verbrennungsmotors
  6. Liebe im Kapitalismus zwischen Geschlechtergleichheit und Marktorientierung
  7. Improved sensorimotor control is not connected with improved proprioception
  8. An analytical approach to evaluating nonmonotonic functions of fuzzy numbers
  9. The Success and Failure of Financial Innovations: The Case of Louis Bachelier
  10. Individuelle mathematische Lernprozesse erfassen, herausfordern und begleiten
  11. Export entry, export exit and productivity in German manufacturing industries
  12. Precipitation Kinetics of AA6082: An Experimental and Numerical Investigation
  13. Capital market imperfections and trade liberalization in general equilibrium
  14. Dimension estimates for certain sets of infinite complex continued fractions
  15. Geometric disturbance decoupling control of vehicles with active suspensions
  16. Abnormal extrusion texture and reversed yield asymmetry in a Mg–Y-Sm-Zn-Zr alloy
  17. Cascade MIMO P-PID Controllers Applied in an Over-actuated Quadrotor Tilt-Rotor
  18. Zur Diskrepanz impliziter und expliziter sicherheitskritischer Einstellungen.
  19. Belief in free will affects causal attributions when judging others’ behavior
  20. A Unified Contextual Bandit Framework for Long- and Short-Term Recommendations
  21. Wie beeinflusst die Kameraperspektive die Beurteilung der Unterrichtsqualität?
  22. Microhardness and in vitro corrosion of heat-treated Mg-Y-Ag biodegradable alloy
  23. Lengthscale-dependent modelling of ductile failure in metallic microstructures
  24. Graph-based Approaches for Analyzing Team Interaction on the Example of Soccer
  25. Dual Kalman Filters Analysis for Interior Permanent Magnet Synchronous Motors
  26. Accuracy Improvement by Artificial Neural Networks in Technical Vision System
  27. Friction analyses in twisted and helical profile extrusion of aluminum alloys
  28. Obstacle Coordinates Transformation from TVS Body-Frame to AGV Navigation-Frame
  29. Schülervorstellungen und sozialwissenschaftliche Vorstellungen über Migration
  30. Motion-decoupled internal force control in grasping with visco-elastic contacts
  31. The Lotka-Volterra Model for Competition Controlled by a Sliding Mode Approach
  32. Similar factors underlie tree abundance in forests in native and alien ranges
  33. Do the global stochastic trends drive the real house prices in OECD countries?
  34. Sustainable management of marine fish stocks by means of sliding mode control
  35. Effects of Mn and Zn solutes on grain refinement of commercial pure magnesium
  36. Early Detection of Faillure in Conveyor Chain Systems by Wireless Sensor Node
  37. Impact of soft law regulation by corporate governance codes on firm valuation
  38. Microstructure refinement by a novel friction-based processing on Mg-Zn-Ca alloy
  39. Number theoretical peculiarities in the dimension theory of dynamical systems