Language Model Transformers as Evaluators for Open-domain Dialogues

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

Computer-based systems for communication with humans are a cornerstone of AI research since the 1950s. So far, the most effective way to assess the quality of the dialogues produced by these systems is to use resource-intensive manual labor instead of automated means. In this work, we investigate whether language models (LM) based on transformer neural networks can indicate the quality of a conversation. In a general sense, language models are methods that learn to predict one or more words based on an already given context. Due to their unsupervised nature, they are candidates for efficient, automatic indication of dialogue quality. We demonstrate that human evaluators have a positive correlation between the output of the language models and scores. We also provide some insights into their behavior and inner-working in a conversational context.

Original languageEnglish
Title of host publicationCOLING 2020 - 28th International Conference on Computational Linguistics : Proceedings of the Conference
EditorsDonia Scott, Nuria Bel, Chengqing Zong
Number of pages12
PublisherAssociation for Computational Linguistics (ACL)
Publication date01.01.2020
Pages6797-6808
ISBN (electronic)9781952148279
DOIs
Publication statusPublished - 01.01.2020
Externally publishedYes
Event28th International Conference on Computational Linguistics, COLING 2020 - Virtual, Online, Spain
Duration: 08.12.202013.12.2020
https://coling2020.org
https://coling2020.org/COLING2020programme.pdf

Bibliographical note

We acknowledge the support of the EU projects Cleopatra (GA 812997) and TAILOR (GA 952215), the Federal Ministry for Economic Affairs and Energy (BMWi) project SPEAKER (FKZ 01MK20011A), the German Federal Ministry of Education and Research (BMBF) projects and excellence clusters ML2R (FKZ 01 15 18038 A/B/C), MLwin (01S18050 D/F), ScaDS.AI (01/S18026A) as well as the Fraunhofer Zukunftsstiftung project JOSEPH.

Publisher Copyright:
© 2020 COLING 2020 - 28th International Conference on Computational Linguistics, Proceedings of the Conference. All rights reserved.

Recently viewed

Publications

  1. Citizen relationship management
  2. A tale of scale: Plot but not neighbourhood tree diversity increases leaf litter ant diversity
  3. A generalized α-level decomposition concept for numerical fuzzy calculus
  4. Calculating the True Profitability of Pollution Prevention
  5. Trust in scientists, risk perception, conspiratorial beliefs, and unrealistic optimism
  6. Mapping Amazon's logistical footprint on the Ruhr
  7. Scientific and local ecological knowledge, shaping perceptions towards protected areas and related ecosystem services
  8. Introduction
  9. Do better pre-migration skills accelerate immigrants’ wage assimilation?
  10. How selective are real wage cuts?
  11. Promoting diversity of thought: bridging knowledge systems for a pluriverse approach to research
  12. Interventionen im Datenraum
  13. What is normal?
  14. rudimentäre Schreibung
  15. Designing an AI Governance Framework
  16. Innovative approaches in mathematical modeling
  17. Separating Cognitive and Content Domains in Mathematical Competence
  18. What can be learnt from the brazilian cerrado?
  19. Acquisitional pragmatics
  20. The influence of a consequence on the readiness potential preceding a self-initiated motor act
  21. Elevated temperature and varied load response of AS41 at bolted joint
  22. An automated, modular system for organic waste utilization using heterotrophic alga Galdieria sulphuraria
  23. Armed to Kill
  24. Internet of Things-Specific Challenges for Enterprise Architectures
  25. Prologue: Analyzing the Fine Details of Political Commitment
  26. Fehler und Versuch. Parteispenden und ihre Regulierung
  27. Proactivity and Adaptability
  28. Management guidelines to address cultural challenges and facilitate values-based innovation through gamification
  29. Processability of Mg-Gd Powder via Friction Extrusion
  30. Turing-Medien
  31. Walking Text and Writing Space
  32. Activity-based working
  33. Direct and Mn-Controlled Indirect Iron Oxidation by Leptothrix discophora SS-1 and Leptothrix cholodnii
  34. Comparison of Reusable and Disposable Laparatomy Pads
  35. Fostering inclusive teaching competences