Discussion on the validity of commonly used reliability indices in sports medicine and exercise science: a critical review with data simulations

Research output: Journal contributionsScientific review articlesResearch

Authors

Apart from objectivity and validity, reliability is considered a precondition for testing within scientific works, as unreliable testing protocols limit conclusions, especially for practical application. Classification guidelines commonly refer to relative reliability, focusing on Pearson correlation coefficients (rp) and intraclass correlation coefficients (ICC). On those, the standard error of measurement (SEM) and the minimal detectable change (MDC) are often calculated in addition to the variability coefficient (CV). These, however, do not account for systematic or random errors (e.g., standardization problems). To illustrate, we applied common reliability statistics in sports science on simulated data which extended the sample size of two original counter-movement-jump sessions from (youth) elite basketball players. These show that excellent rp and ICC (≥ 0.9) without a systematic bias were accompanied by a mean absolute percentage error of over 20%. Furthermore, we showed that the ICC does not account for systematic errors and has only limited value for accuracy, which can cause misleading conclusions of data. While a simple re-organization of data caused an improvement in relative reliability and reduced limits of agreement meaningfully, systematic errors occurred. This example underlines the lack of validity and objectivity of commonly used ICC-based reliability statistics (SEM, MDC) to quantify the primary and secondary variance sources. After revealing several caveats in the literature (e.g., neglecting of the systematic and random error or not distinguishing between protocol and device reliability), we suggest a methodological approach to provide reliable data collections as a precondition for valid conclusions by, e.g., recommending pre-set acceptable measurement errors.
Original languageEnglish
Article numbere0216065
JournalEuropean Journal of Applied Physiology
Volume125
Issue number6
Pages (from-to)1511-1526
Number of pages16
ISSN1439-6319
DOIs
Publication statusPublished - 06.2025

Bibliographical note

Publisher Copyright:
© The Author(s) 2025.

Recently viewed

Researchers

  1. Jörg Terhechte (✝)

Publications

  1. States and traits
  2. Study of digital morphing tools in the architectural design process
  3. Towards a thick understanding of sustainability transitions - Linking transition management, capabilities and social practices
  4. Synthesis of Room-Temperature Ionic Liquids with the Weakly Coordinating [Al(ORF)(4)](-) Anion (R-F = C(H)(CF3)(2)) and the Determination of Their Principal Physical Properties
  5. Self-efficacy, self-regulation and mathematics performance of competitive junior rowers vs. regular students
  6. Designing a global energy system based on 100% renewables for 2050 - GENeSYS-MOD
  7. Does participatory governance help address long-term environmental problems?
  8. Multilayer Material System Analysis of wind turbines
  9. Part IV: Theoretical and methodological advancements in disturbance rejection and robust control
  10. Lufthansa Cargo´s alliance strategy: Spinning a web of stable relations
  11. Do it again
  12. PyFin-sentiment
  13. The need for transnational networks and transdisciplinary education for sustainable development in UNESCO Biosphere Reserves in the Global South
  14. Changing the Rules
  15. Jurisdiction and applicable law in cases of damage from space in Europe - The advent of the most suitable choice - Rome II
  16. Does ESG performance have an impact on financial performance?
  17. A highly endangered species on the edge
  18. Vom Sagbaren zum Machbaren?
  19. Dealing with availability and response expectations: Are older employees at an advantage and why?
  20. Jointly experimenting for transformation?
  21. New trends in pragmatics
  22. Vibration Converter with Passive Energy Management for Battery‐Less Wireless Sensor Nodes in Predictive Maintenance
  23. Advanced Controlling - eine Ideenskizze
  24. The importance of understanding the multiple dimensions of power in stakeholder participation for effective biodiversity conservation
  25. Effect of salinity on growth of mussels, Mytilus edulis, with special reference to Great Belt (Denmark)
  26. Towards a Deconstruction of the Screen