Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: An observational study

Research output: Journal contributionsJournal articlesResearchpeer-review

Authors

  • Tom E. Hardwicke
  • Manuel Bohn
  • Kyle MacDonald
  • Emily Hembacher
  • Michèle B. Nuijten
  • Benjamin N. Peloquin
  • Benjamin E. Demayo
  • Bria Long
  • Erica J. Yoon
  • Michael C. Frank

For any scientific report, repeating the original analyses upon the original data should yield the original outcomes. We evaluated analytic reproducibility in 25 Psychological Science articles awarded open data badges between 2014 and 2015. Initially, 16 (64%, 95% confidence interval [43,81]) articles contained at least one 'major numerical discrepancy' (>10% difference) prompting us to request input from original authors. Ultimately, target values were reproducible without author involvement for 9 (36% [20,59]) articles; reproducible with author involvement for 6 (24% [8,47]) articles; not fully reproducible with no substantive author response for 3 (12% [0,35]) articles; and not fully reproducible despite author involvement for 7 (28% [12,51]) articles. Overall, 37 major numerical discrepancies remained out of 789 checked values (5% [3,6]), but original conclusions did not appear affected. Non-reproducibility was primarily caused by unclear reporting of analytic procedures. These results highlight that open data alone is not sufficient to ensure analytic reproducibility.

Original languageEnglish
Article number201494
JournalRoyal Society Open Science
Volume8
Issue number1
Number of pages9
ISSN2054-5703
DOIs
Publication statusPublished - 06.01.2021
Externally publishedYes

Bibliographical note

Funding Information:
Data accessibility. All data, materials and analysis scripts related to this study are publicly available on the Open Science Framework (https://osf.io/n3dej/). To facilitate reproducibility, this manuscript was written by interleaving regular prose and analysis code using knitr [28] and papaja [29], and is available in a Code Ocean container (https://doi.org/ 10.24433/CO.1796004.v3) which re-creates the software environment in which the original analyses were performed. Authors’ contributions. T.E.H. and M.C.F. designed the study. T.E.H., M.B., K.M., E.H., M.B.N., B.N.P., B.E.d.M., B.L., E.J.Y. and M.C.F. conducted the reproducibility checks. T.E.H. performed the data analysis. T.E.H. and M.C.F. wrote the manuscript. M.B. and M.B.N. provided feedback on the manuscript. All authors gave final approval for publication. Competing interests. We declare we have no competing interests. Funding. T.E.H.’s contribution was enabled by a general support grant awarded to the Meta-Research Innovation Center at Stanford (METRICS) from the Laura and John Arnold Foundation and a grant from the Einstein Foundation and Stiftung Charité awarded to the Meta-Research Innovation Center Berlin (METRIC-B). Acknowledgements. We are grateful to the authors of the original articles for their assistance with the reproducibility checks. We thank students from Stanford’s Psych 251 class, who contributed to the initial reproducibility checks.

Publisher Copyright:
© 2021 The Authors.

    Research areas

  • journal policy, meta-research, open badges, open data, open science, reproducibility
  • Psychology

Documents

DOI

Recently viewed

Researchers

  1. Tilottama Biswas

Publications

  1. An Interactive Layers Model of Self-Regulated Learning and Cognitive Load
  2. Methodological support for the selection of simplified equations of state for modeling technical fluids
  3. Lexical markers of common grounds
  4. New and Rapid Fully Automated Method for Determination of Tazobactam and Piperacillin in Fatty Tissue and Serum by Column-Switching Liquid Chromatography
  5. Assessing the Bonding Interface Characteristics and Mechanical Properties of Bobbin Tool Friction Stir Welded Dissimilar Aluminum Alloy Joints
  6. Systematic learning in water governance: insights from five local adaptive management projects for water quality innovation
  7. The promise and Pitfalls of a blended, video- and coaching-based professional development program in Germany
  8. General strategies to increase the repeatability in non-target screening by liquid chromatography-high resolution mass spectrometry
  9. Spaces with a temper
  10. Are survey expectations theory-consistent?
  11. Distinguishing between struggling and skilled readers based on their prosodic speech patterns in oral reading
  12. Introduction to the Handbook on the Politics of Taxation
  13. Transformative Indikatoren
  14. Stress corrosion of the Mg-Zn-Zr alloy system using C-ring tests
  15. Profilanalyse
  16. Reconciling Deliberateness and Emergence in Digital Transformation Strategy Formation — An Organizational Improvisation Perspective
  17. Entwicklung von Instrumenten zur Vermeidung von Lebensmittelabfällen
  18. Indigenous and local knowledge in environmental management for human-nature connectedness
  19. The multipole resonance probe: kinetic damping in its spectra
  20. Age and gender composition of the workforce, productivity and profits