Automated scoring in the era of artificial intelligence: An empirical study with Turkish essays

Publikation: Beiträge in ZeitschriftenZeitschriftenaufsätzeForschungbegutachtet

Authors

Automated scoring (AS) has gained significant attention as a tool to enhance the efficiency and reliability of assessment processes. Yet, its application in under-represented languages, such as Turkish, remains limited. This study addresses this gap by empirically evaluating AS for Turkish using a zero-shot approach with a rubric powered by OpenAI's GPT-4o. A dataset of 590 essays written by learners of Turkish as a second language was scored by professional human raters and an artificial intelligence (AI) model integrated via a custom-built interface. The scoring rubric, grounded in the Common European Framework of Reference for Languages, assessed six dimensions of writing quality. Results revealed a strong alignment between human and AI scores with a Quadratic Weighted Kappa of 0.72, Pearson correlation of 0.73, and an overlap measure of 83.5 %. Analysis of rater effects showed minimal influence on score discrepancies, though factors such as experience and gender exhibited modest effects. These findings demonstrate the potential of AI-driven scoring in Turkish, offering valuable insights for broader implementation in under-represented languages, such as the possible source of disagreements between human and AI scores. Conclusions from a specific writing task with a single human rater underscore the need for future research to explore diverse inputs and multiple raters.

OriginalspracheEnglisch
Aufsatznummer103784
ZeitschriftSystem
Jahrgang133
Anzahl der Seiten12
ISSN0346-251X
DOIs
PublikationsstatusErschienen - 10.2025

Bibliographische Notiz

Publisher Copyright:
© 2025 The Authors

DOI

Zuletzt angesehen

Forschende

  1. Gereon Wellmann

Aktivitäten

  1. On the perception and effectiveness of the feedback quality from a digital learning platform
  2. International Convention of Psychological Science 2017
  3. Knowledge of result versus elaborated feedback: Students‘ perception of feedback on a digital learning platform
  4. Scene as Ecosystem, Scenes as Parts of Ecosystems or Scene versus Ecosystem? Some considerations about the compability of two conceptional approaches
  5. Artificial Intelligence in Criminal Law
  6. A mobile phone supported internet-based intervention for depressive symptoms in diabetes mellitus type 1 and type 2: design of a randomized controlled trial
  7. From Left to Right: Shifts in Political Hegemony Against the Backdrop of Structural Transformations of Capitalism and Class Composition
  8. OR for children: Lego robotic Warehouse Simulation
  9. Art and Sustainability: Aesthetics of Complexity
  10. Project-Based Education and Other Activating Strategies in Science Education 2020
  11. University of Illinois
  12. 5th Int. Summer Academy „Energy and the Environment“ 2008
  13. LCE2016
  14. Vis-à-Vis
  15. 6th Institute of Electrical and Electronics Engineers International Conference on Modelling, Identification and Control - 2014
  16. The forest beyond the trees: a network perspective on governing nature's contributions to people co-production
  17. Arbeitsgemeinschaft Simulation (ASIM) Fachtagung 2017
  18. Workshop on the Exploration of Low Temperature plasma Physics - WELTPP 2018
  19. Revisiting the evolution of strategic initiatives
  20. Nachhaltiger Konsum in der Wachstumsgesellschaft
  21. The Leuphana Bachelor: The structure of the degree programme and the Major "Studium Individuale"