Modeling items for text comprehension assessment using confirmatory factor analysis

Research output: Journal contributionsJournal articlesResearchpeer-review

Standard

Modeling items for text comprehension assessment using confirmatory factor analysis. / Tschense, Monika; Wallot, Sebastian.
In: Frontiers in Psychology, Vol. 13, 966347, 20.10.2022.

Research output: Journal contributionsJournal articlesResearchpeer-review

Harvard

APA

Vancouver

Bibtex

@article{050726ef87d04ccd85bf3046de9a2c00,
title = "Modeling items for text comprehension assessment using confirmatory factor analysis",
abstract = "Reading is a complex cognitive task with the ultimate goal of comprehending the written input. For longer, connected text, readers generate a mental representation that serves as its basis. Due to limited cognitive resources, common models of discourse representation assume distinct processing levels, each relying on different processing mechanisms. However, only little research addresses distinct representational levels when text comprehension is assessed, analyzed or modelled. Moreover, current studies that tried to relate process measures of reading (e.g., reading times, eye movements) to comprehension did not consider comprehension as a multi-faceted, but rather a uni-dimensional construct, usually assessed with one-shot items. Thus, the first aim of this paper is to use confirmatory factor analysis (CFA) to test whether comprehension can be modelled as a uni-or multi-dimensional concept. The second aim is to investigate how well widely used one-shot items can be used to capture comprehension. 400 participants read one of three short stories of comparable length, linguistic characteristics, and complexity. Based on the evaluation of three independent raters per story, 16 wh-questions and 60 yes/no-statements were compiled in order to retrieve information at micro and inference level, and 16 main contents were extracted to capture information at the macro level in participants{\textquoteright} summaries. Still, only a fraction of these items showed satisfactory psychometric properties and factor loadings – a blatant result considering the common practice for item selection. For CFA, two models were set up that address text comprehension as either a one-dimensional construct (a uni-factor model with a single comprehension factor), or a three-dimensional construct reflecting the three distinct representational levels (three correlated first-order factors). Across stories and item types, model fit was consistently better for the three-factor model providing evidence for a multi-dimensional construct of text comprehension. Our results provide concrete guidance for the preparation of comprehension measurements in studies investigating the reading process.",
keywords = "comprehension assessment, discourse representation, mental model, reading, reading comprehension, text comprehension, Psychology",
author = "Monika Tschense and Sebastian Wallot",
note = "Funding Information: The study was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) by grants to SW (project numbers 397523278 and 442405852). Publisher Copyright: Copyright {\textcopyright} 2022 Tschense and Wallot.",
year = "2022",
month = oct,
day = "20",
doi = "10.3389/fpsyg.2022.966347",
language = "English",
volume = "13",
journal = "Frontiers in Psychology",
issn = "1664-1078",
publisher = "Frontiers Media SA",

}

RIS

TY - JOUR

T1 - Modeling items for text comprehension assessment using confirmatory factor analysis

AU - Tschense, Monika

AU - Wallot, Sebastian

N1 - Funding Information: The study was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) by grants to SW (project numbers 397523278 and 442405852). Publisher Copyright: Copyright © 2022 Tschense and Wallot.

PY - 2022/10/20

Y1 - 2022/10/20

N2 - Reading is a complex cognitive task with the ultimate goal of comprehending the written input. For longer, connected text, readers generate a mental representation that serves as its basis. Due to limited cognitive resources, common models of discourse representation assume distinct processing levels, each relying on different processing mechanisms. However, only little research addresses distinct representational levels when text comprehension is assessed, analyzed or modelled. Moreover, current studies that tried to relate process measures of reading (e.g., reading times, eye movements) to comprehension did not consider comprehension as a multi-faceted, but rather a uni-dimensional construct, usually assessed with one-shot items. Thus, the first aim of this paper is to use confirmatory factor analysis (CFA) to test whether comprehension can be modelled as a uni-or multi-dimensional concept. The second aim is to investigate how well widely used one-shot items can be used to capture comprehension. 400 participants read one of three short stories of comparable length, linguistic characteristics, and complexity. Based on the evaluation of three independent raters per story, 16 wh-questions and 60 yes/no-statements were compiled in order to retrieve information at micro and inference level, and 16 main contents were extracted to capture information at the macro level in participants’ summaries. Still, only a fraction of these items showed satisfactory psychometric properties and factor loadings – a blatant result considering the common practice for item selection. For CFA, two models were set up that address text comprehension as either a one-dimensional construct (a uni-factor model with a single comprehension factor), or a three-dimensional construct reflecting the three distinct representational levels (three correlated first-order factors). Across stories and item types, model fit was consistently better for the three-factor model providing evidence for a multi-dimensional construct of text comprehension. Our results provide concrete guidance for the preparation of comprehension measurements in studies investigating the reading process.

AB - Reading is a complex cognitive task with the ultimate goal of comprehending the written input. For longer, connected text, readers generate a mental representation that serves as its basis. Due to limited cognitive resources, common models of discourse representation assume distinct processing levels, each relying on different processing mechanisms. However, only little research addresses distinct representational levels when text comprehension is assessed, analyzed or modelled. Moreover, current studies that tried to relate process measures of reading (e.g., reading times, eye movements) to comprehension did not consider comprehension as a multi-faceted, but rather a uni-dimensional construct, usually assessed with one-shot items. Thus, the first aim of this paper is to use confirmatory factor analysis (CFA) to test whether comprehension can be modelled as a uni-or multi-dimensional concept. The second aim is to investigate how well widely used one-shot items can be used to capture comprehension. 400 participants read one of three short stories of comparable length, linguistic characteristics, and complexity. Based on the evaluation of three independent raters per story, 16 wh-questions and 60 yes/no-statements were compiled in order to retrieve information at micro and inference level, and 16 main contents were extracted to capture information at the macro level in participants’ summaries. Still, only a fraction of these items showed satisfactory psychometric properties and factor loadings – a blatant result considering the common practice for item selection. For CFA, two models were set up that address text comprehension as either a one-dimensional construct (a uni-factor model with a single comprehension factor), or a three-dimensional construct reflecting the three distinct representational levels (three correlated first-order factors). Across stories and item types, model fit was consistently better for the three-factor model providing evidence for a multi-dimensional construct of text comprehension. Our results provide concrete guidance for the preparation of comprehension measurements in studies investigating the reading process.

KW - comprehension assessment

KW - discourse representation

KW - mental model

KW - reading

KW - reading comprehension

KW - text comprehension

KW - Psychology

UR - http://www.scopus.com/inward/record.url?scp=85141185437&partnerID=8YFLogxK

U2 - 10.3389/fpsyg.2022.966347

DO - 10.3389/fpsyg.2022.966347

M3 - Journal articles

C2 - 36337503

AN - SCOPUS:85141185437

VL - 13

JO - Frontiers in Psychology

JF - Frontiers in Psychology

SN - 1664-1078

M1 - 966347

ER -

Documents

DOI

Recently viewed

Publications

  1. Recurrence Quantification Analysis of Processes and Products of Discourse
  2. Lessons learned for spatial modelling of ecosystem services in support of ecosystem accounting
  3. Construct Objectification and De-Objectification in Organization Theory
  4. Computational modeling of amorphous polymers
  5. Modeling and numerical simulation of multiscale behavior in polycrystals via extended crystal plasticity
  6. Influence of Process Parameters and Die Design on the Microstructure and Texture Development of Direct Extruded Magnesium Flat Products
  7. Simple saturated PID control for fast transient of motion systems
  8. Dynamic Lot Size Optimization with Reinforcement Learning
  9. The delay vector variance method and the recurrence quantification analysis of energy markets
  10. Introducing parametric uncertainty into a nonlinear friction model
  11. Faulty Process Detection Using Machine Learning Techniques
  12. TextGraphs 2024 Shared Task on Text-Graph Representations for Knowledge Graph Question Answering
  13. Clause identification using entropy guided transformation learning
  14. Mathematical Modeling for Robot 3D Laser Scanning in Complete Darkness Environments to Advance Pipeline Inspection
  15. Dispatching rule selection with Gaussian processes
  16. Constraints are the solution, not the problem
  17. Dynamic priority based dispatching of AGVs in flexible job shops
  18. Mining positional data streams
  19. Understanding the properties of isospectral points and pairs in graphs
  20. Improving students’ science text comprehension through metacognitive self-regulation when applying learning strategies
  21. Comments on "Tracking Control of Robotic Manipulators With Uncertain Kinematics and Dynamics"
  22. Computing regression statistics from grouped data
  23. From Knowledge to Application
  24. Gaussian processes for dispatching rule selection in production scheduling