Joint Item Response Models for Manual and Automatic Scores on Open-Ended Test Items

Publikation: Beiträge in ZeitschriftenZeitschriftenaufsätzeForschungbegutachtet

Authors

Test items using open-ended response formats can increase an instrument’s construct validity. However, traditionally, their application in educational testing requires human coders to score the responses. Manual scoring not only increases operational costs but also prohibits the use of evidence from open-ended items to inform routing decisions in adaptive designs. Using machine learning and natural language processing, automatic scoring provides classifiers that can instantly assign scores to text responses. Although optimized for agreement with manual scores, automatic scoring is not perfectly accurate and introduces an additional source of error into the response process, leading to a misspecification of the measurement model used with the manual score. We propose two joint models for manual and automatic scores of automatically scored open-ended items. Our models extend a given model from Item Response Theory for the manual scores by a component for the automatic scores, accounting for classification errors. The models were evaluated using data from the Programme for International Student Assessment (2012) and simulated data, demonstrating their capacity to mitigate the impact of classification errors on ability estimation compared to a baseline that disregards classification errors.

OriginalspracheEnglisch
ZeitschriftPsychometrika
ISSN0033-3123
DOIs
PublikationsstatusAngenommen/Im Druck - 2025

Bibliographische Notiz

Publisher Copyright:
© 2025 Cambridge University Press. All rights reserved.

DOI

Zuletzt angesehen

Aktivitäten

  1. Automatic Detection and Classification of State Heads and Common People?
  2. Closing Session: Summary Notes
  3. A Lyapunov based PI controller with an anti-windup scheme for a purification process of potable water
  4. Combining an Internal SMC with an External MTPA Control Loop for an Interior PMSM
  5. The influence of polycentricity on collaborative environmental management – the case of EU Water Framework Directive implementation in Germany
  6. Commitment Strategies for Sustainability: How Corporations Can Create Value through New Governance
  7. Blyton’s Island(s)
  8. How stakeholder characteristics influence the perception and evaluation of CSR communication: a mixed-method approach to communication reception
  9. Modeling Self-Organization (3rd International Conference of the ESHS)
  10. Balancing Acts
  11. Carbon Dioxide Treatment, Summary and Presentation of the Final Version of the Computerprogram CO2
  12. Splinternet and globalisation: Two early models of internet opposed
  13. Creating transdisciplinary research spaces for sustainable development
  14. Rethinking Gamification: A Critical Approach to Gamification
  15. Mutual Learning and Knowledge Integration in Transdisciplinary Development Teams: Empirical Findings about a Collaborative Format in Teacher Education
  16. Self-tuning of a kalman filter applied in a DC drive and in a kalman-based sensor
  17. Organizational Practices for the Aging Workforce: Validation of an English Version of the Later Life Workplace Index
  18. Grenzflächen der Informatik - 2006
  19. Experiences on the theme of actions for sustainable development in the field of educational systems
  20. DCRLectures Summer Semester 2016
  21. 2021 3rd International Conference on Soft Computing and its Engineering Applications
  22. Universität Ulm
  23. 42nd Joint Sessions of Workshops - ECPR 2014

Publikationen

  1. "And I Think That Is a Very Straightforward Way of Dealing With It''
  2. Applications of the Simultaneous Modular Approach in the Field of Material Flow Analysis
  3. Enacting migration through data practices
  4. Combining an Internal SMC with an External MTPA Control Loop for an Interior PMSM
  5. Reading Comprehension as Embodied Action: Exploratory Findings on Nonlinear Eye Movement Dynamics and Comprehension of Scientific Texts
  6. Novel Class B Amplifier-Based Inductive Charging System for Wireless Sensor Nodes
  7. Using measures of reading time regularity (RTR) to quantify eye movement dynamics, and how they are shaped by linguistic information
  8. Combining fusion-based and solid-state additive manufacturing
  9. German Utilities and Distributed PV
  10. An interdisciplinary methodological guide for quantifying associations between ecosystem services
  11. Modeling of temperature- and strain-driven intermetallic compound evolution in an Al-Mg system via a multiphase-field approach with application to refill friction stir spot welding
  12. Analysis of the relevance of models, influencing factors and the point in time of the forecast on the prediction quality in order-related delivery time determination using machine learning
  13. Integration of demand forecasts in ABC-XYZ analysis
  14. CubeQA—question answering on RDF data cubes
  15. Business Analytics and Making Decision Based on Kalman Filter in Stock Prediction Case
  16. Earnings Less Risk-Free Interest Charge (ERIC) and Stock Returns—A Value-Based Management Perspective on ERIC’s Relative and Incremental Information Content