Language Model Transformers as Evaluators for Open-domain Dialogues

Publikation: Beiträge in SammelwerkenAufsätze in KonferenzbändenForschungbegutachtet

Authors

Computer-based systems for communication with humans are a cornerstone of AI research since the 1950s. So far, the most effective way to assess the quality of the dialogues produced by these systems is to use resource-intensive manual labor instead of automated means. In this work, we investigate whether language models (LM) based on transformer neural networks can indicate the quality of a conversation. In a general sense, language models are methods that learn to predict one or more words based on an already given context. Due to their unsupervised nature, they are candidates for efficient, automatic indication of dialogue quality. We demonstrate that human evaluators have a positive correlation between the output of the language models and scores. We also provide some insights into their behavior and inner-working in a conversational context.

OriginalspracheEnglisch
TitelCOLING 2020 - 28th International Conference on Computational Linguistics : Proceedings of the Conference
HerausgeberDonia Scott, Nuria Bel, Chengqing Zong
Anzahl der Seiten12
VerlagAssociation for Computational Linguistics (ACL)
Erscheinungsdatum01.01.2020
Seiten6797-6808
ISBN (elektronisch)9781952148279
DOIs
PublikationsstatusErschienen - 01.01.2020
Extern publiziertJa
Veranstaltung28th International Conference on Computational Linguistics, COLING 2020 - Virtual, Online, Spanien
Dauer: 08.12.202013.12.2020
https://coling2020.org
https://coling2020.org/COLING2020programme.pdf

Bibliographische Notiz

Funding Information:
We acknowledge the support of the EU projects Cleopatra (GA 812997) and TAILOR (GA 952215), the Federal Ministry for Economic Affairs and Energy (BMWi) project SPEAKER (FKZ 01MK20011A), the German Federal Ministry of Education and Research (BMBF) projects and excellence clusters ML2R (FKZ 01 15 18038 A/B/C), MLwin (01S18050 D/F), ScaDS.AI (01/S18026A) as well as the Fraunhofer Zukunftsstiftung project JOSEPH.

Publisher Copyright:
© 2020 COLING 2020 - 28th International Conference on Computational Linguistics, Proceedings of the Conference. All rights reserved.

DOI

Zuletzt angesehen

Aktivitäten

  1. Discursive Depoliticisation? Communicative Dissonance in the Age of Populism
  2. Leuphana-Forschungskonferenz - 2013
  3. The politics of uncertainty
  4. Politics of volition or constraint?
  5. Mythos und Wahrheit
  6. Workshop: "Transfer of Tourism Knowledge and Tourism Results – Publishing Issues of the Future" - 2006
  7. Lehrerfortbildung 2012
  8. Hölderlin's Theory of Tragedy
  9. Long-term Internships of Physical Education Pre-service Teachers in the Pandemic: A Mixed-methods Anaylsis
  10. Pierre Bourdieu im ästhetischen Feld
  11. Guidelines for Assessing Biodiversity-Related Risks and Opportunities in the Financial Sector: An Aid for Implementing the VfU Biodiversity Principles
  12. exMNU-Tagung 2008
  13. European University, Florenz
  14. THE FUTURE OF ephemera - 2019
  15. Identifying Global Challenges for Future Tourism and Tourism Management
  16. Transportation Research Part C (Zeitschrift)
  17. GEW-Leuphana-Bildungstag 2012
  18. Underground Clubs in Transition. Scene based Music Venues and Spatial Strategies in Hamburg St. Pauli.
  19. Hölderlins philosophische Entwicklung
  20. Transcultural Memory and Ethics
  21. 16th (EC)² Conference - 2005
  22. Not terminated: In a Plessnerian perspective ‘cyborgized’ men still remain ‘human’ beings
  23. Impulse für die Projektforschung 2008
  24. “Doing Justice! Doing Just This! Understanding justice in transdisciplinary and transformative research”: LRN Summer School
  25. HyperKult 20 - Trivialisierung 2011
  26. Goethe Graduate Academy
  27. Peace Science and Public Policy (Zeitschrift)
  28. Association for Information Systems (AIS) (Externe Organisation)