Reporting and Analysing the Environmental Impact of Language Models on the Example of Commonsense Question Answering with External Knowledge

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearch

Authors

Human-produced emissions are growing at an alarming rate, causing already observable changes in the climate and environment in general. Each year global carbon dioxide emissions hit a new record, and it is reported that 0.5% of total US greenhouse gas emissions are attributed to data centres as of 2021. The release of ChatGPT in late 2022 sparked social interest in Large Language Models (LLMs), the new generation of Language Models with a large number of parameters and trained on massive amounts of data. Currently, numerous companies are releasing products featuring various LLMs, with many more models in development and awaiting release. Deep Learning research is a competitive field, with only models that reach top performance attracting attention and being utilized. Hence, achieving better accuracy and results is often the first priority, while the model's efficiency and the environmental impact of the study are neglected. However, LLMs demand substantial computational resources and are very costly to train, both financially and environmentally. It becomes essential to raise awareness and promote conscious decisions about algorithmic and hardware choices. Providing information on training time, the approximate carbon dioxide emissions and power consumption would assist future studies in making necessary adjustments and determining the compatibility of available computational resources with model requirements. In this study, we infused T5 LLM with external knowledge and fine-tuned the model for Question-Answering task. Furthermore, we calculated and reported the approximate environmental impact for both steps. The findings demonstrate that the smaller models may not always be sustainable options, and increased training does not always imply better performance. The most optimal outcome is achieved by carefully considering both performance and efficiency factors.
Original languageEnglish
Title of host publicationSustainable AI Conference 2023: Sustainable AI Across Borders : Conference Proceedings
Volumeabs/2408.01453
DOIs
Publication statusIn preparation - 2024
Event2. Sustainable AI Conference 2023: Sustainable AI Across Borders - University of Bonn, Bonn, Germany
Duration: 30.05.202301.06.2023
Conference number: 2
https://www.uni-bonn.de/de/veranstaltungen/sustainable-ai-conference-2023-sustainable-ai-across-borders

Recently viewed

Activities

  1. Surviving in the wild: determinants and practices of strategic initiative development
  2. European Academy of Management (Externe Organisation)
  3. Validity of a mathematics test for the selection of university applicants for teacher training
  4. On the perception and effectiveness of the feedback quality from a digital learning platform
  5. International Convention of Psychological Science 2017
  6. Knowledge of result versus elaborated feedback: Students‘ perception of feedback on a digital learning platform
  7. Scene as Ecosystem, Scenes as Parts of Ecosystems or Scene versus Ecosystem? Some considerations about the compability of two conceptional approaches
  8. Artificial Intelligence in Criminal Law
  9. A mobile phone supported internet-based intervention for depressive symptoms in diabetes mellitus type 1 and type 2: design of a randomized controlled trial
  10. From Left to Right: Shifts in Political Hegemony Against the Backdrop of Structural Transformations of Capitalism and Class Composition
  11. OR for children: Lego robotic Warehouse Simulation
  12. Art and Sustainability: Aesthetics of Complexity
  13. Project-Based Education and Other Activating Strategies in Science Education 2020
  14. University of Illinois
  15. 5th Int. Summer Academy „Energy and the Environment“ 2008
  16. LCE2016
  17. Vis-à-Vis
  18. 6th Institute of Electrical and Electronics Engineers International Conference on Modelling, Identification and Control - 2014
  19. The forest beyond the trees: a network perspective on governing nature's contributions to people co-production
  20. Arbeitsgemeinschaft Simulation (ASIM) Fachtagung 2017
  21. Workshop on the Exploration of Low Temperature plasma Physics - WELTPP 2018