Language Model Transformers as Evaluators for Open-domain Dialogues

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

Computer-based systems for communication with humans are a cornerstone of AI research since the 1950s. So far, the most effective way to assess the quality of the dialogues produced by these systems is to use resource-intensive manual labor instead of automated means. In this work, we investigate whether language models (LM) based on transformer neural networks can indicate the quality of a conversation. In a general sense, language models are methods that learn to predict one or more words based on an already given context. Due to their unsupervised nature, they are candidates for efficient, automatic indication of dialogue quality. We demonstrate that human evaluators have a positive correlation between the output of the language models and scores. We also provide some insights into their behavior and inner-working in a conversational context.

Original languageEnglish
Title of host publicationCOLING 2020 - 28th International Conference on Computational Linguistics : Proceedings of the Conference
EditorsDonia Scott, Nuria Bel, Chengqing Zong
Number of pages12
PublisherAssociation for Computational Linguistics (ACL)
Publication date01.01.2020
Pages6797-6808
ISBN (electronic)9781952148279
DOIs
Publication statusPublished - 01.01.2020
Externally publishedYes
Event28th International Conference on Computational Linguistics, COLING 2020 - Virtual, Online, Spain
Duration: 08.12.202013.12.2020
https://coling2020.org
https://coling2020.org/COLING2020programme.pdf

Bibliographical note

We acknowledge the support of the EU projects Cleopatra (GA 812997) and TAILOR (GA 952215), the Federal Ministry for Economic Affairs and Energy (BMWi) project SPEAKER (FKZ 01MK20011A), the German Federal Ministry of Education and Research (BMBF) projects and excellence clusters ML2R (FKZ 01 15 18038 A/B/C), MLwin (01S18050 D/F), ScaDS.AI (01/S18026A) as well as the Fraunhofer Zukunftsstiftung project JOSEPH.

Publisher Copyright:
© 2020 COLING 2020 - 28th International Conference on Computational Linguistics, Proceedings of the Conference. All rights reserved.

Recently viewed

Projects

  1. StadtRaumKlang

Publications

  1. Plant traits alone are poor predictors of ecosystem properties and long-term ecosystem functioning
  2. Understanding Context Collapse for Social Media Users
  3. Horizontal, but not vertical canopy structure is related to stand functional diversity in a subtropical slope forest
  4. Explaining Age and Gender Differences in Employment Rates
  5. Recurring patterns and blueprints of industrial symbioses as structural units for an it tool
  6. Modeling of microstructural pattern formation in crystal plasticity
  7. Ludic interfaces
  8. Induced Technological Change: Exploring its Implications for the Economics of Atmospheric Stabilization
  9. Trust in scientists, risk perception, conspiratorial beliefs, and unrealistic optimism
  10. Structure as Infrastructure: The Interrelation of Fiber and Construction
  11. On the Epistemology of Computer Simulation
  12. Effects of strategy instructions on learning from text and pictures
  13. States of Comparability
  14. Measurement of cognitive load in multimedia learning
  15. Implementierung eines Fehlerpräventionsprogramms für gefahrenintensive Arbeitsprozesse
  16. Who wants to take an intelligence test? Personality and achievement motivation in the context of ability testing
  17. Repeated sampling detects gene flow in a flightless ground beetle in a fragmented landscape
  18. Concept Maps in der Hochschullehre
  19. Multiple
  20. Impact of prescribed burning on the nutrient balance of heathlands with particular reference to nitrogen and phosphorus
  21. Inequality in the Transition from Primary to Secondary School
  22. University, failed
  23. On-board pneumatic pressure generation methods for soft robotics applications
  24. Doing statistics, enacting the nation
  25. The Use of Media in Intercultural Dialogue "dialogo_dialog"!
  26. What can be learnt from the brazilian cerrado?