Modeling items for text comprehension assessment using confirmatory factor analysis

Research output: Journal contributionsJournal articlesResearchpeer-review

Authors

Reading is a complex cognitive task with the ultimate goal of comprehending the written input. For longer, connected text, readers generate a mental representation that serves as its basis. Due to limited cognitive resources, common models of discourse representation assume distinct processing levels, each relying on different processing mechanisms. However, only little research addresses distinct representational levels when text comprehension is assessed, analyzed or modelled. Moreover, current studies that tried to relate process measures of reading (e.g., reading times, eye movements) to comprehension did not consider comprehension as a multi-faceted, but rather a uni-dimensional construct, usually assessed with one-shot items. Thus, the first aim of this paper is to use confirmatory factor analysis (CFA) to test whether comprehension can be modelled as a uni-or multi-dimensional concept. The second aim is to investigate how well widely used one-shot items can be used to capture comprehension. 400 participants read one of three short stories of comparable length, linguistic characteristics, and complexity. Based on the evaluation of three independent raters per story, 16 wh-questions and 60 yes/no-statements were compiled in order to retrieve information at micro and inference level, and 16 main contents were extracted to capture information at the macro level in participants’ summaries. Still, only a fraction of these items showed satisfactory psychometric properties and factor loadings – a blatant result considering the common practice for item selection. For CFA, two models were set up that address text comprehension as either a one-dimensional construct (a uni-factor model with a single comprehension factor), or a three-dimensional construct reflecting the three distinct representational levels (three correlated first-order factors). Across stories and item types, model fit was consistently better for the three-factor model providing evidence for a multi-dimensional construct of text comprehension. Our results provide concrete guidance for the preparation of comprehension measurements in studies investigating the reading process.

Original languageEnglish
Article number966347
JournalFrontiers in Psychology
Volume13
Number of pages10
ISSN1664-1078
DOIs
Publication statusPublished - 20.10.2022

Bibliographical note

Funding Information:
The study was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) by grants to SW (project numbers 397523278 and 442405852).

Publisher Copyright:
Copyright © 2022 Tschense and Wallot.

    Research areas

  • comprehension assessment, discourse representation, mental model, reading, reading comprehension, text comprehension
  • Psychology

Documents

DOI

Recently viewed

Publications

  1. Supporting the Development and Implementation of a Digitalization Strategy in SMEs through a Lightweight Architecture-based Method
  2. Dynamic environment modelling and prediction for autonomous systems
  3. A tutorial introduction to adaptive fractal analysis
  4. Fast, Fully Automated Analysis of Voriconazole from Serum by LC-LC-ESI-MS-MS with Parallel Column-Switching Technique
  5. Robust Control of Mobile Transportation Object with 3D Technical Vision System
  6. Eighth Workshop on Mining and Learning with Graphs
  7. Knowledge Graph Question Answering and Large Language Models
  8. Introduction Mobile Digital Practices. Situating People, Things, and Data
  9. What can conservation strategies learn from the ecosystem services approach?
  10. A Class of Simple Stochastic Online Bin Packing Algorithms
  11. Constraints are the solution, not the problem
  12. Overcoming Multi-legacy Application Challenges through Building Dynamic Capabilities for Low-Code Adoption
  13. Positioning Improvement for a Laser Scanning System using cSORPD control
  14. Determinants and Outcomes of Dual Distribution:
  15. Handicaps in job assignment
  16. Covert and overt automatic imitation are correlated
  17. Exploring large vegetation databases to detect temporal trends in species occurrences
  18. Trait-based approaches to analyze links between the drivers of change and ecosystem services
  19. When, Where, and How Nature Matters for Ecosystem Services
  20. Lessons learned and challenges for environmental management in Colombia
  21. A latent state-trait analysis of current achievement motivation across different tasks of cognitive ability
  22. An interdisciplinary perspective on scaling in transitions
  23. Increased Reliability of Draw-In Prediction in a Single Stage Deep-Drawing Operation via Transfer Learning
  24. Steering of land use in the context of sustainable development