Reporting and Analysing the Environmental Impact of Language Models on the Example of Commonsense Question Answering with External Knowledge

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearch

Authors

Human-produced emissions are growing at an alarming rate, causing already observable changes in the climate and environment in general. Each year global carbon dioxide emissions hit a new record, and it is reported that 0.5% of total US greenhouse gas emissions are attributed to data centres as of 2021. The release of ChatGPT in late 2022 sparked social interest in Large Language Models (LLMs), the new generation of Language Models with a large number of parameters and trained on massive amounts of data. Currently, numerous companies are releasing products featuring various LLMs, with many more models in development and awaiting release. Deep Learning research is a competitive field, with only models that reach top performance attracting attention and being utilized. Hence, achieving better accuracy and results is often the first priority, while the model's efficiency and the environmental impact of the study are neglected. However, LLMs demand substantial computational resources and are very costly to train, both financially and environmentally. It becomes essential to raise awareness and promote conscious decisions about algorithmic and hardware choices. Providing information on training time, the approximate carbon dioxide emissions and power consumption would assist future studies in making necessary adjustments and determining the compatibility of available computational resources with model requirements. In this study, we infused T5 LLM with external knowledge and fine-tuned the model for Question-Answering task. Furthermore, we calculated and reported the approximate environmental impact for both steps. The findings demonstrate that the smaller models may not always be sustainable options, and increased training does not always imply better performance. The most optimal outcome is achieved by carefully considering both performance and efficiency factors.
Original languageEnglish
Title of host publicationSustainable AI Conference 2023: Sustainable AI Across Borders : Conference Proceedings
Volumeabs/2408.01453
DOIs
Publication statusIn preparation - 2024
Event2. Sustainable AI Conference 2023: Sustainable AI Across Borders - University of Bonn, Bonn, Germany
Duration: 30.05.202301.06.2023
Conference number: 2
https://www.uni-bonn.de/de/veranstaltungen/sustainable-ai-conference-2023-sustainable-ai-across-borders

Recently viewed

Activities

  1. Knowledge Spaces
  2. Workshop „Meta-Image Day 2012”
  3. Liquidity, Flows, Circulation: The Cultural Logic of Environmentalization (2nd part) 2021
  4. Language Learning in Blended-Learning Projects: Moodle, Web 2.0, and Learner Agency
  5. Ars Electronica
  6. Blogs in the Foreign Language Classroom
  7. 9th International Multi-Conference on Systems, Signals and Devices - SSD 2012
  8. Are Self-Employed Time and Money Poor? Dynamics of Interpendent Multidimensional Poverty with German Time Use Diary Data
  9. Developing the ‘Benign by Design’ Approach for a Rational Design of Green Derivatives of b -Blockers: Propranolol as an Example
  10. From Christiane to Elisabeth. The 19th Century Genesis of the Intellectually Working Woman and the Epistemological Dependency on Structures of Desire in Hegel and Nietzsche
  11. Institutional dynamics of affecting and being affected: The emotionalization of injustice and the threat of withdrawing the organizational identification
  12. Scene as Ecosystem, Scenes as Parts of Ecosystems or Scene versus Ecosystem? Some considerations about the compability of two conceptional approaches
  13. 24th IEEE International Conference on Business Informatics
  14. 13th Trends in Enterprise Architecture Research Workshop - TEAR 2018
  15. Exploring Sustainability in Virtual Space
  16. Towards a sustainable Southern Transylvania: Recognizing existing contributions to reach sustainable visions and empowering stakeholders
  17. 1st International Conference of the International Association for Computing and Philosophy - IACAP 2011
  18. Going Green 2015 - Exploring Sustainability in Virtual Space