Modeling items for text comprehension assessment using confirmatory factor analysis

Publikation: Beiträge in ZeitschriftenZeitschriftenaufsätzeForschungbegutachtet

Standard

Modeling items for text comprehension assessment using confirmatory factor analysis. / Tschense, Monika; Wallot, Sebastian.
in: Frontiers in Psychology, Jahrgang 13, 966347, 20.10.2022.

Publikation: Beiträge in ZeitschriftenZeitschriftenaufsätzeForschungbegutachtet

Harvard

APA

Vancouver

Bibtex

@article{050726ef87d04ccd85bf3046de9a2c00,
title = "Modeling items for text comprehension assessment using confirmatory factor analysis",
abstract = "Reading is a complex cognitive task with the ultimate goal of comprehending the written input. For longer, connected text, readers generate a mental representation that serves as its basis. Due to limited cognitive resources, common models of discourse representation assume distinct processing levels, each relying on different processing mechanisms. However, only little research addresses distinct representational levels when text comprehension is assessed, analyzed or modelled. Moreover, current studies that tried to relate process measures of reading (e.g., reading times, eye movements) to comprehension did not consider comprehension as a multi-faceted, but rather a uni-dimensional construct, usually assessed with one-shot items. Thus, the first aim of this paper is to use confirmatory factor analysis (CFA) to test whether comprehension can be modelled as a uni-or multi-dimensional concept. The second aim is to investigate how well widely used one-shot items can be used to capture comprehension. 400 participants read one of three short stories of comparable length, linguistic characteristics, and complexity. Based on the evaluation of three independent raters per story, 16 wh-questions and 60 yes/no-statements were compiled in order to retrieve information at micro and inference level, and 16 main contents were extracted to capture information at the macro level in participants{\textquoteright} summaries. Still, only a fraction of these items showed satisfactory psychometric properties and factor loadings – a blatant result considering the common practice for item selection. For CFA, two models were set up that address text comprehension as either a one-dimensional construct (a uni-factor model with a single comprehension factor), or a three-dimensional construct reflecting the three distinct representational levels (three correlated first-order factors). Across stories and item types, model fit was consistently better for the three-factor model providing evidence for a multi-dimensional construct of text comprehension. Our results provide concrete guidance for the preparation of comprehension measurements in studies investigating the reading process.",
keywords = "comprehension assessment, discourse representation, mental model, reading, reading comprehension, text comprehension, Psychology",
author = "Monika Tschense and Sebastian Wallot",
note = "Funding Information: The study was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) by grants to SW (project numbers 397523278 and 442405852). Publisher Copyright: Copyright {\textcopyright} 2022 Tschense and Wallot.",
year = "2022",
month = oct,
day = "20",
doi = "10.3389/fpsyg.2022.966347",
language = "English",
volume = "13",
journal = "Frontiers in Psychology",
issn = "1664-1078",
publisher = "Frontiers Media SA",

}

RIS

TY - JOUR

T1 - Modeling items for text comprehension assessment using confirmatory factor analysis

AU - Tschense, Monika

AU - Wallot, Sebastian

N1 - Funding Information: The study was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) by grants to SW (project numbers 397523278 and 442405852). Publisher Copyright: Copyright © 2022 Tschense and Wallot.

PY - 2022/10/20

Y1 - 2022/10/20

N2 - Reading is a complex cognitive task with the ultimate goal of comprehending the written input. For longer, connected text, readers generate a mental representation that serves as its basis. Due to limited cognitive resources, common models of discourse representation assume distinct processing levels, each relying on different processing mechanisms. However, only little research addresses distinct representational levels when text comprehension is assessed, analyzed or modelled. Moreover, current studies that tried to relate process measures of reading (e.g., reading times, eye movements) to comprehension did not consider comprehension as a multi-faceted, but rather a uni-dimensional construct, usually assessed with one-shot items. Thus, the first aim of this paper is to use confirmatory factor analysis (CFA) to test whether comprehension can be modelled as a uni-or multi-dimensional concept. The second aim is to investigate how well widely used one-shot items can be used to capture comprehension. 400 participants read one of three short stories of comparable length, linguistic characteristics, and complexity. Based on the evaluation of three independent raters per story, 16 wh-questions and 60 yes/no-statements were compiled in order to retrieve information at micro and inference level, and 16 main contents were extracted to capture information at the macro level in participants’ summaries. Still, only a fraction of these items showed satisfactory psychometric properties and factor loadings – a blatant result considering the common practice for item selection. For CFA, two models were set up that address text comprehension as either a one-dimensional construct (a uni-factor model with a single comprehension factor), or a three-dimensional construct reflecting the three distinct representational levels (three correlated first-order factors). Across stories and item types, model fit was consistently better for the three-factor model providing evidence for a multi-dimensional construct of text comprehension. Our results provide concrete guidance for the preparation of comprehension measurements in studies investigating the reading process.

AB - Reading is a complex cognitive task with the ultimate goal of comprehending the written input. For longer, connected text, readers generate a mental representation that serves as its basis. Due to limited cognitive resources, common models of discourse representation assume distinct processing levels, each relying on different processing mechanisms. However, only little research addresses distinct representational levels when text comprehension is assessed, analyzed or modelled. Moreover, current studies that tried to relate process measures of reading (e.g., reading times, eye movements) to comprehension did not consider comprehension as a multi-faceted, but rather a uni-dimensional construct, usually assessed with one-shot items. Thus, the first aim of this paper is to use confirmatory factor analysis (CFA) to test whether comprehension can be modelled as a uni-or multi-dimensional concept. The second aim is to investigate how well widely used one-shot items can be used to capture comprehension. 400 participants read one of three short stories of comparable length, linguistic characteristics, and complexity. Based on the evaluation of three independent raters per story, 16 wh-questions and 60 yes/no-statements were compiled in order to retrieve information at micro and inference level, and 16 main contents were extracted to capture information at the macro level in participants’ summaries. Still, only a fraction of these items showed satisfactory psychometric properties and factor loadings – a blatant result considering the common practice for item selection. For CFA, two models were set up that address text comprehension as either a one-dimensional construct (a uni-factor model with a single comprehension factor), or a three-dimensional construct reflecting the three distinct representational levels (three correlated first-order factors). Across stories and item types, model fit was consistently better for the three-factor model providing evidence for a multi-dimensional construct of text comprehension. Our results provide concrete guidance for the preparation of comprehension measurements in studies investigating the reading process.

KW - comprehension assessment

KW - discourse representation

KW - mental model

KW - reading

KW - reading comprehension

KW - text comprehension

KW - Psychology

UR - http://www.scopus.com/inward/record.url?scp=85141185437&partnerID=8YFLogxK

U2 - 10.3389/fpsyg.2022.966347

DO - 10.3389/fpsyg.2022.966347

M3 - Journal articles

C2 - 36337503

AN - SCOPUS:85141185437

VL - 13

JO - Frontiers in Psychology

JF - Frontiers in Psychology

SN - 1664-1078

M1 - 966347

ER -

Dokumente

DOI

Zuletzt angesehen

Aktivitäten

  1. Project Workshop on "Worker Flows, Match Quality, and Productivity" - 2019
  2. 13th IFAC and IEEE Conference on Programmable Devices and Embedded Systems - PDeS 2015
  3. Applied Econometrics with Stata for PhD Students
  4. Princeton University
  5. Thinking of Time - A Resource which Should be Allocated Equally
  6. Changing learning environments at university? Comparing the learning strategies of non-traditional European students engaged in lifelong learning.
  7. Predicting negotiation success with a multitude of negotiators’ inter-individual differences—a latent personality model of the successful negotiator
  8. Visualizing and analyzing big data sets: Results from the Student Bodies-Eating Disorders study
  9. "Crowds and Party" Reading Workshop with Jody Dean - 2019
  10. European University Institute
  11. Where is language use in the description of the Englishes? - ESSE 2006
  12. Comfort and Adaptive Cruise Control in Highly Automated Vehicles
  13. Panel Cointegration Testing with Time Trend and Analysis of Money Demand in OECD Countries
  14. Navigating Educational Ambidexterity: Exploring Leader-Member Exchange and Open Innovation
  15. Digital Capitalism meets Leberkaspeppi: Temporal Orientations in Business Models as a Source of Platform Power in Mature Industries
  16. THE EUROPEAN CONFERENCE ON MACHINE LEARNING & PRINCIPLES AND PRACTICE OF KNOWLEDGE DISCOVERY IN DATABASES 2017
  17. Ghosting the City – Zooming in on Otherwise Publics in Virtual Worlds
  18. 6th Workshop on Unintended Consequences
  19. Internet-based guided self-help to reduce depressive symptoms in teachers: Results from a randomized controlled trial
  20. Modelling biodegradability based on OECD 301D data for the design of mineralising ionic liquids
  21. Mathematical and Computational Applications (Fachzeitschrift)
  22. Response to Philip Hogh: Suffering, Pain and the Idea of Progress

Publikationen

  1. Cue predictability changes scaling in eye-movement fluctuations
  2. Service Level Driven Stock Allocation
  3. Digging into the roots
  4. Design for Product Care—Development of Design Strategies and a Toolkit for Sustainable Consumer Behaviour
  5. The effect of structural complexity on large mammal occurrence in revegetation
  6. Towards Advanced Learning in Dispatching Rule-Based Scheuling
  7. Integrating inductive and deductive analysis to identify and characterize archetypical social-ecological systems and their changes
  8. "Introduction," communication +1
  9. What Makes for a Good Theory? How to Evaluate a Theory Using the Strength Model of Self-Control as an Example
  10. Embedding Evidence on Conservation Interventions Within a Context of Multilevel Governance
  11. Drafts in Action
  12. Development and comparison of processing maps of Mg-3Sn-1Ca alloy from data obtained in tension versus compression
  13. Modeling of temperature- and strain-driven intermetallic compound evolution in an Al-Mg system via a multiphase-field approach with application to refill friction stir spot welding
  14. Discriminative clustering for market segmentation
  15. From Open Access to Open Science
  16. How to move the transition to sustainable food consumption towards a societal tipping point
  17. Integrating business models and enterprise architecture
  18. Concepts, Formats, and Methods of Participation
  19. Effective digital practice in the competence-oriented English as a foreign language classroom in Germany
  20. Trajectory tracking using MPC and a velocity observer for flat actuator systems in automotive applications