Joint Item Response Models for Manual and Automatic Scores on Open-Ended Test Items

Publikation: Beiträge in ZeitschriftenZeitschriftenaufsätzeForschungbegutachtet

Authors

Test items using open-ended response formats can increase an instrument’s construct validity. However, traditionally, their application in educational testing requires human coders to score the responses. Manual scoring not only increases operational costs but also prohibits the use of evidence from open-ended items to inform routing decisions in adaptive designs. Using machine learning and natural language processing, automatic scoring provides classifiers that can instantly assign scores to text responses. Although optimized for agreement with manual scores, automatic scoring is not perfectly accurate and introduces an additional source of error into the response process, leading to a misspecification of the measurement model used with the manual score. We propose two joint models for manual and automatic scores of automatically scored open-ended items. Our models extend a given model from Item Response Theory for the manual scores by a component for the automatic scores, accounting for classification errors. The models were evaluated using data from the Programme for International Student Assessment (2012) and simulated data, demonstrating their capacity to mitigate the impact of classification errors on ability estimation compared to a baseline that disregards classification errors.

OriginalspracheEnglisch
ZeitschriftPsychometrika
ISSN0033-3123
DOIs
PublikationsstatusAngenommen/Im Druck - 2025

Bibliographische Notiz

Publisher Copyright:
© 2025 Cambridge University Press. All rights reserved.

DOI

Zuletzt angesehen

Publikationen

  1. Always on Call: Is There an Age Advantage in Dealing with Availability and Response Expectations?
  2. Modeling and simulation of the microstructural behaviour in thermal sprayed coatings
  3. Metrics for Experimentation Programs: Categories, Benefits and Challenges
  4. General management principles and a checklist of strategies to guide forest biodiversity conservation
  5. An integrative research framework for enabling transformative adaptation
  6. Art 160: Powers and functions
  7. Normative Integration of the Avantgarde?
  8. Effectiveness of a gratitude app at reducing repetitive negative thinking as a transdiagnostic risk factor in the general population
  9. Temporal changes in taxonomic and functional alpha and beta diversity across tree communities in subtropical Atlantic forests
  10. Future Challenges for Global Tourism
  11. Telearbeit in Deutschland
  12. Neuro-Esthetics : mapological foundations and applications (map 2003)
  13. Technikvergessenheit?
  14. Moving beyond the heuristic of creative destruction
  15. An existential perspective on the psychological function of shamans
  16. The "Attention" Entrapment Phenomenon
  17. Deficits in Emotion-Regulation Skills Predict Alcohol Use During and After Cognitive Behavioral Therapy for Alcohol Dependence
  18. § 29 Windenergie
  19. Emissions of decamethylcyclopentasiloxane from Chicago
  20. Residual stresses of the as-cast Mg-xCa alloys with hot sprues by neutron diffraction
  21. Interrogating the city
  22. Einen gemeinsamen Code finden
  23. Transitions to plant-based diets
  24. Natural vs. financial insurances in the management of public good ecosystems
  25. Emergency Politics After Globalization
  26. Minisymposium: Dynamische Visualisierung in der Lehre von Mathematik
  27. Immune cells contribute to myelin degeneration and axonopathic changes in mice overexpressing proteolipid protein in oligodendrocytes
  28. “Have you felt angry lately?”
  29. Helsingør statement on poly- and perfluorinated alkyl substances (PFASs)
  30. Continuous pretreatment, hydrolysis, and fermentation of organic residues for the production of biochemicals
  31. Actor-Network Theory II
  32. Home range and habitat use by the pacas (Cuniculus paca) in a montane tropical forest in Bolivia
  33. Do unbiased people act more rationally? - The case of comparative realism and vaccine intention
  34. Selbstbild und Selbstvertrauen
  35. Influence of spectrally selective solar cells on microalgae growth in photo-bioreactors
  36. Do high incomes reflect individual performance?
  37. Make it your Break! Benefits of Person-Break Fit for Post-Break Affect