Modeling items for text comprehension assessment using confirmatory factor analysis

Research output: Journal contributionsJournal articlesResearchpeer-review

Standard

Modeling items for text comprehension assessment using confirmatory factor analysis. / Tschense, Monika; Wallot, Sebastian.
In: Frontiers in Psychology, Vol. 13, 966347, 20.10.2022.

Research output: Journal contributionsJournal articlesResearchpeer-review

Harvard

APA

Vancouver

Bibtex

@article{050726ef87d04ccd85bf3046de9a2c00,
title = "Modeling items for text comprehension assessment using confirmatory factor analysis",
abstract = "Reading is a complex cognitive task with the ultimate goal of comprehending the written input. For longer, connected text, readers generate a mental representation that serves as its basis. Due to limited cognitive resources, common models of discourse representation assume distinct processing levels, each relying on different processing mechanisms. However, only little research addresses distinct representational levels when text comprehension is assessed, analyzed or modelled. Moreover, current studies that tried to relate process measures of reading (e.g., reading times, eye movements) to comprehension did not consider comprehension as a multi-faceted, but rather a uni-dimensional construct, usually assessed with one-shot items. Thus, the first aim of this paper is to use confirmatory factor analysis (CFA) to test whether comprehension can be modelled as a uni-or multi-dimensional concept. The second aim is to investigate how well widely used one-shot items can be used to capture comprehension. 400 participants read one of three short stories of comparable length, linguistic characteristics, and complexity. Based on the evaluation of three independent raters per story, 16 wh-questions and 60 yes/no-statements were compiled in order to retrieve information at micro and inference level, and 16 main contents were extracted to capture information at the macro level in participants{\textquoteright} summaries. Still, only a fraction of these items showed satisfactory psychometric properties and factor loadings – a blatant result considering the common practice for item selection. For CFA, two models were set up that address text comprehension as either a one-dimensional construct (a uni-factor model with a single comprehension factor), or a three-dimensional construct reflecting the three distinct representational levels (three correlated first-order factors). Across stories and item types, model fit was consistently better for the three-factor model providing evidence for a multi-dimensional construct of text comprehension. Our results provide concrete guidance for the preparation of comprehension measurements in studies investigating the reading process.",
keywords = "comprehension assessment, discourse representation, mental model, reading, reading comprehension, text comprehension, Psychology",
author = "Monika Tschense and Sebastian Wallot",
note = "Funding Information: The study was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) by grants to SW (project numbers 397523278 and 442405852). Publisher Copyright: Copyright {\textcopyright} 2022 Tschense and Wallot.",
year = "2022",
month = oct,
day = "20",
doi = "10.3389/fpsyg.2022.966347",
language = "English",
volume = "13",
journal = "Frontiers in Psychology",
issn = "1664-1078",
publisher = "Frontiers Media SA",

}

RIS

TY - JOUR

T1 - Modeling items for text comprehension assessment using confirmatory factor analysis

AU - Tschense, Monika

AU - Wallot, Sebastian

N1 - Funding Information: The study was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) by grants to SW (project numbers 397523278 and 442405852). Publisher Copyright: Copyright © 2022 Tschense and Wallot.

PY - 2022/10/20

Y1 - 2022/10/20

N2 - Reading is a complex cognitive task with the ultimate goal of comprehending the written input. For longer, connected text, readers generate a mental representation that serves as its basis. Due to limited cognitive resources, common models of discourse representation assume distinct processing levels, each relying on different processing mechanisms. However, only little research addresses distinct representational levels when text comprehension is assessed, analyzed or modelled. Moreover, current studies that tried to relate process measures of reading (e.g., reading times, eye movements) to comprehension did not consider comprehension as a multi-faceted, but rather a uni-dimensional construct, usually assessed with one-shot items. Thus, the first aim of this paper is to use confirmatory factor analysis (CFA) to test whether comprehension can be modelled as a uni-or multi-dimensional concept. The second aim is to investigate how well widely used one-shot items can be used to capture comprehension. 400 participants read one of three short stories of comparable length, linguistic characteristics, and complexity. Based on the evaluation of three independent raters per story, 16 wh-questions and 60 yes/no-statements were compiled in order to retrieve information at micro and inference level, and 16 main contents were extracted to capture information at the macro level in participants’ summaries. Still, only a fraction of these items showed satisfactory psychometric properties and factor loadings – a blatant result considering the common practice for item selection. For CFA, two models were set up that address text comprehension as either a one-dimensional construct (a uni-factor model with a single comprehension factor), or a three-dimensional construct reflecting the three distinct representational levels (three correlated first-order factors). Across stories and item types, model fit was consistently better for the three-factor model providing evidence for a multi-dimensional construct of text comprehension. Our results provide concrete guidance for the preparation of comprehension measurements in studies investigating the reading process.

AB - Reading is a complex cognitive task with the ultimate goal of comprehending the written input. For longer, connected text, readers generate a mental representation that serves as its basis. Due to limited cognitive resources, common models of discourse representation assume distinct processing levels, each relying on different processing mechanisms. However, only little research addresses distinct representational levels when text comprehension is assessed, analyzed or modelled. Moreover, current studies that tried to relate process measures of reading (e.g., reading times, eye movements) to comprehension did not consider comprehension as a multi-faceted, but rather a uni-dimensional construct, usually assessed with one-shot items. Thus, the first aim of this paper is to use confirmatory factor analysis (CFA) to test whether comprehension can be modelled as a uni-or multi-dimensional concept. The second aim is to investigate how well widely used one-shot items can be used to capture comprehension. 400 participants read one of three short stories of comparable length, linguistic characteristics, and complexity. Based on the evaluation of three independent raters per story, 16 wh-questions and 60 yes/no-statements were compiled in order to retrieve information at micro and inference level, and 16 main contents were extracted to capture information at the macro level in participants’ summaries. Still, only a fraction of these items showed satisfactory psychometric properties and factor loadings – a blatant result considering the common practice for item selection. For CFA, two models were set up that address text comprehension as either a one-dimensional construct (a uni-factor model with a single comprehension factor), or a three-dimensional construct reflecting the three distinct representational levels (three correlated first-order factors). Across stories and item types, model fit was consistently better for the three-factor model providing evidence for a multi-dimensional construct of text comprehension. Our results provide concrete guidance for the preparation of comprehension measurements in studies investigating the reading process.

KW - comprehension assessment

KW - discourse representation

KW - mental model

KW - reading

KW - reading comprehension

KW - text comprehension

KW - Psychology

UR - http://www.scopus.com/inward/record.url?scp=85141185437&partnerID=8YFLogxK

U2 - 10.3389/fpsyg.2022.966347

DO - 10.3389/fpsyg.2022.966347

M3 - Journal articles

C2 - 36337503

AN - SCOPUS:85141185437

VL - 13

JO - Frontiers in Psychology

JF - Frontiers in Psychology

SN - 1664-1078

M1 - 966347

ER -

Documents

DOI

Recently viewed

Publications

  1. A Lean Convolutional Neural Network for Vehicle Classification
  2. An extended analytical approach to evaluating monotonic functions of fuzzy numbers
  3. Effects of diversity versus segregation on automatic approach and avoidance behavior towards own and other ethnic groups
  4. Developing a Complex Portrait of Content Teaching for Multilingual Learners via Nonlinear Theoretical Understandings
  5. Support vector machines with example dependent costs
  6. The structure of emotions in learning situations
  7. Reciprocal Relationships Between Dispositional Optimism and Work Experiences
  8. Conceptualizing Role Development in Agile Transformations
  9. Predicate‐based model of problem‐solving for robotic actions planning
  10. Vergütung, variable
  11. Sensitivity to complexity - an important prerequisite of problem solving mathematics teaching
  12. Study of fuzzy controllers performance
  13. Performance Saga: Interview 01
  14. Employing A-B tests for optimizing prices levels in e-commerce applications
  15. An analytical approach to evaluating nonmonotonic functions of fuzzy numbers
  16. Between Recognition and Abstraction
  17. Species constancy depends on plot size - A problem for vegetation classification and how it can be solved
  18. The Relation of Children's Performances in Spatial Tasks at Two Different Scales of Space
  19. Petri net based EMIS-mappers for flexible manufacturing systems
  20. Schooling, local knowledge and working memory
  21. Practice and carryover effects when using small interaction devices
  22. Scaling-based Least Squares Methods with Implemented Kalman filter Approach for Nano-Parameters Identification
  23. Fusion of knowledge bases for better navigation of wheeled mobile robotic group with 3D TVS
  24. Learning linear classifiers sensitive to example dependent and noisy costs
  25. Model and Validation of the Electromagnetic Interference Produced by Power Transmission Lines in Robotic Systems
  26. Cognitive load in reading a foreign language text with multimedia aids and the influence of verbal and spatial abilities
  27. Mapping the Order of New Migration
  28. Geometric analysis of a laser scanner functioning based on dynamic triangulation
  29. Challenging the status quo of accelerator research: Concluding remarks