Language Model Transformers as Evaluators for Open-domain Dialogues

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

Computer-based systems for communication with humans are a cornerstone of AI research since the 1950s. So far, the most effective way to assess the quality of the dialogues produced by these systems is to use resource-intensive manual labor instead of automated means. In this work, we investigate whether language models (LM) based on transformer neural networks can indicate the quality of a conversation. In a general sense, language models are methods that learn to predict one or more words based on an already given context. Due to their unsupervised nature, they are candidates for efficient, automatic indication of dialogue quality. We demonstrate that human evaluators have a positive correlation between the output of the language models and scores. We also provide some insights into their behavior and inner-working in a conversational context.

Original languageEnglish
Title of host publicationCOLING 2020 - 28th International Conference on Computational Linguistics : Proceedings of the Conference
EditorsDonia Scott, Nuria Bel, Chengqing Zong
Number of pages12
PublisherAssociation for Computational Linguistics (ACL)
Publication date01.01.2020
Pages6797-6808
ISBN (electronic)9781952148279
DOIs
Publication statusPublished - 01.01.2020
Externally publishedYes
Event28th International Conference on Computational Linguistics, COLING 2020 - Virtual, Online, Spain
Duration: 08.12.202013.12.2020
https://coling2020.org
https://coling2020.org/COLING2020programme.pdf

Bibliographical note

We acknowledge the support of the EU projects Cleopatra (GA 812997) and TAILOR (GA 952215), the Federal Ministry for Economic Affairs and Energy (BMWi) project SPEAKER (FKZ 01MK20011A), the German Federal Ministry of Education and Research (BMBF) projects and excellence clusters ML2R (FKZ 01 15 18038 A/B/C), MLwin (01S18050 D/F), ScaDS.AI (01/S18026A) as well as the Fraunhofer Zukunftsstiftung project JOSEPH.

Publisher Copyright:
© 2020 COLING 2020 - 28th International Conference on Computational Linguistics, Proceedings of the Conference. All rights reserved.

Recently viewed

Publications

  1. Understanding Context Collapse for Social Media Users
  2. Transparency in an Age of Digitalization and Responsibility
  3. Low Resource Question Answering: An Amharic Benchmarking Dataset
  4. Heterogenität
  5. Are the terms “Socio-economic status” and “Class status” a warped form of reasoning for Max Weber?
  6. Microsatellites and allozymes as the genetic memory of habitat fragmentation and defragmentation in populations of the ground beetle Carabus auronitens (Col., Carabidae)
  7. A practical perspective on repatriate knowledge transfer
  8. Semi-polar root exudates in natural grassland communities
  9. Sol-gel technology for greener and more sustainable antimicrobial textiles that use silica matrices with C, and Ag and ZnO as biocides
  10. Fallstudie
  11. Actor analysis as a tool for exploring the decision-making processes in environmental governance
  12. Carbocyclic cis-[1.1.1]-tris-σ-homobenzenes - Syntheses by triple epoxide → cyclopropane conversions, structural data, [σ2s+σ2s+σ2s] cycloreversions
  13. Interventionen im Datenraum
  14. Time and Income Poverty – An Interdependent Multidimensional Poverty Approach with German Time Use Diary Data
  15. Plants, Androids and Operators
  16. The Use of Media in Intercultural Dialogue "dialogo_dialog"!
  17. SAP exchange infrastructure for developers
  18. Fallstudie
  19. Customer Orientation of Service Employees—Toward a Conceptual Framework of a Key Relationship Marketing Construct
  20. Study of digital morphing tools in the architectural design process
  21. Does the introduction of the Euro have an effect on subjective hypotheses about the price-quality relationship?
  22. Performance Saga: Interview 03
  23. External State-Building and Why Norms Matter
  24. Combined experimental–numerical study on residual stresses induced by a single impact as elementary process of mechanical peening
  25. Conceptual frameworks and methods for advancing invasion ecology
  26. Welteis