Language Model Transformers as Evaluators for Open-domain Dialogues

Publikation: Beiträge in SammelwerkenAufsätze in KonferenzbändenForschungbegutachtet

Authors

Computer-based systems for communication with humans are a cornerstone of AI research since the 1950s. So far, the most effective way to assess the quality of the dialogues produced by these systems is to use resource-intensive manual labor instead of automated means. In this work, we investigate whether language models (LM) based on transformer neural networks can indicate the quality of a conversation. In a general sense, language models are methods that learn to predict one or more words based on an already given context. Due to their unsupervised nature, they are candidates for efficient, automatic indication of dialogue quality. We demonstrate that human evaluators have a positive correlation between the output of the language models and scores. We also provide some insights into their behavior and inner-working in a conversational context.

OriginalspracheEnglisch
TitelCOLING 2020 - 28th International Conference on Computational Linguistics : Proceedings of the Conference
HerausgeberDonia Scott, Nuria Bel, Chengqing Zong
Anzahl der Seiten12
VerlagAssociation for Computational Linguistics (ACL)
Erscheinungsdatum01.01.2020
Seiten6797-6808
ISBN (elektronisch)9781952148279
DOIs
PublikationsstatusErschienen - 01.01.2020
Extern publiziertJa
Veranstaltung28th International Conference on Computational Linguistics, COLING 2020 - Virtual, Online, Spanien
Dauer: 08.12.202013.12.2020
https://coling2020.org
https://coling2020.org/COLING2020programme.pdf

Bibliographische Notiz

Funding Information:
We acknowledge the support of the EU projects Cleopatra (GA 812997) and TAILOR (GA 952215), the Federal Ministry for Economic Affairs and Energy (BMWi) project SPEAKER (FKZ 01MK20011A), the German Federal Ministry of Education and Research (BMBF) projects and excellence clusters ML2R (FKZ 01 15 18038 A/B/C), MLwin (01S18050 D/F), ScaDS.AI (01/S18026A) as well as the Fraunhofer Zukunftsstiftung project JOSEPH.

Publisher Copyright:
© 2020 COLING 2020 - 28th International Conference on Computational Linguistics, Proceedings of the Conference. All rights reserved.

DOI

Zuletzt angesehen

Publikationen

  1. Contested World Order
  2. The use of intellectual capital as a competitive tool
  3. Transparency in an Age of Digitalization and Responsibility
  4. Introduction
  5. Collisions in space
  6. Knowledge acquisition and development in sustainability-oriented small and medium-sized enterprises
  7. Optimization of transport flow on two paths with respect to the passengers time costs
  8. Use of the concept of Bildung in the international science education literature, its potential, and implications for teaching and learning
  9. Form and Relation
  10. A Note on Smoking Behavior and Health Risk Taking
  11. Das Ethos reiner Fraulichkeit
  12. Geochemical Assessment of Sediment Quality Using Multivariate Statistical Analysis of Ennore Creek, North of Chennai, SE Coast of India.
  13. European and national law in history and future
  14. Harmony at the Workplace
  15. Article 3 Universal Application
  16. Identitätspolitik als Strategie der Entprivilegierung
  17. Interregional flows of multiple ecosystem services through global trade in wild species
  18. Is a severe clinical profile an effect modifier in a web-based depression treatment for adults with type 1 or type 2 diabetes ?
  19. Assessing Printability Maps in Additive Manufacturing of Metal Alloys
  20. Symbolische Politik oder echter Einfluss?
  21. A meta-analysis of the contribution of eye movements in processing emotional memories
  22. Green Finance
  23. Polizei und Gewalt
  24. [Paul Celan und] Martin Heidegger
  25. Maschinen – Sprachen
  26. Immanentism
  27. Ästhetische Bildung der Differenz
  28. Introduction
  29. Refugee social work as remote EU border control? Externalization policies and social work in Niger
  30. Notting Hill Gate 4 Basic
  31. Lively Artifacts
  32. Supply Chain Management in wachsenden Märkten
  33. Eine informationsökonomische Analyse des Handwerks
  34. Figuren der Teilhabe in Mischa Kuballs NEW POTT
  35. Forschung zu Energiewende und Partizipation
  36. Zur Analyse des Willens an seinen Randzonen
  37. Napoleon
  38. Politische Strategie
  39. Guest editorial: Leadership in school health promotion. The multiple perspectives of a neglected research area
  40. Polarisierung der Einkommen von Selbständigen?
  41. The Impact of Peer Presence on Cheating
  42. „Ghostly Embodiments“
  43. Das unechte Unterlassungsdelikt
  44. The Visual Turn in Business Management. Film, Graphical Devices and the Consulting Industry
  45. Temporality
  46. How much psychotherapy is needed to treat depression?