Proxy Indicators for the Quality of Open-domain Dialogues

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

The automatic evaluation of open-domain dialogues remains a largely unsolved challenge. Thus, despite the abundance of work done in the field, human judges have to evaluate dialogues' quality. As a consequence, performing such evaluations at scale is usually expensive. This work investigates using a deep-learning model trained on the General Language Understanding Evaluation (GLUE) benchmark to serve as a quality indication of open-domain dialogues. The aim is to use the various GLUE tasks as different perspectives on judging the quality of conversation, thus reducing the need for additional training data or responses that serve as quality references. Due to this nature, the method can infer various quality metrics and derive a component-based overall score. We achieve statistically significant correlation coefficients of up to 0.7.

Original languageEnglish
Title of host publicationEMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings
EditorsMarie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Number of pages22
PublisherAssociation for Computational Linguistics (ACL)
Publication date01.01.2021
Pages7834-7855
ISBN (electronic)9781955917094
DOIs
Publication statusPublished - 01.01.2021
Externally publishedYes
Event2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021 - ONLINE, Punta Cana, Dominican Republic
Duration: 07.11.202111.11.2021
https://2021.emnlp.org

Bibliographical note

Publisher Copyright:
© 2021 Association for Computational Linguistics

Recently viewed

Publications

  1. Unveiling local knowledge
  2. Pathways of Data-driven Business Model Design and Realization
  3. Offline question answering over linked data using limited resources
  4. Geodesign as a boundary management process
  5. Life Cycle Assessment of Consumption Patterns – Understanding the links between changing social practices and environmental impacts
  6. Consequences of extreme weather events for developing countries based on the example of Mongolia
  7. Creating Value from in-Vehicle Data
  8. Challenges for biodiversity monitoring using citizen science in transitioning social-ecological systems
  9. Operationalization of the concept of sustainable development on different time scales
  10. Performance incentives in activity-based management
  11. The impact of explicit references in computer supported collaborative learning: Evidence from eye movement analyses
  12. Employing A-B tests for optimizing prices levels in e-commerce applications
  13. Integrating teacher and student workspaces in a technology-enhanced mathematics lecture
  14. Multi-view hidden markov perceptrons
  15. Exploring the dark and unexpected sides of digitalization
  16. Tschick
  17. Probabilistic movement models and zones of control
  18. Decision-making models for Robotic Warehouse
  19. One step forward, two steps back
  20. Performance Saga: Interview 06
  21. A PD Fuzzy Control of a Nonholonomic Car-Like Robot for Drive Assistant Systems
  22. Integrating multiple elements of environmental justice into urban blue space planning using public participation geographic information systems
  23. Sustainable use of ecosystem services under multiple risks
  24. Children's interpretation of ambiguous pronouns based on prior discourse
  25. Organizational practices for the aging workforce
  26. Conditionality of EU funds: an instrument to enforce EU fundamental values?
  27. The micro-processes during repatriate knowledge transfer
  28. Utilization of protein-rich residues in biotechnological processes
  29. Pathways to Implementation: Evidence on How Participation in Environmental Governance Impacts on Environmental Outcomes
  30. Quantifying ecosystem services of rewetted peatlands − the MoorFutures methodologies
  31. Learning Analytics
  32. The Role of Assessment and Quality Management in Transformations towards Sustainable Development
  33. To help or not to help an outgroup member
  34. Mathematics-specific motivations for choosing a mathematics teaching degree study programme
  35. Top-down biological motion perception does not differ between adults scoring high versus low on autism traits
  36. Soil carbon sequestration
  37. Utilizing Synchrotron Radiation for Phase Identification in Mg Alloys
  38. Learning Analytics an Hochschulen
  39. Standing up against Discrimination and Exclusion
  40. CALPHAD-based modeling of pressure-dependent Al, Cu and Li unary systems