Crowdsourcing Hypothesis Tests: Making transparent how design choices shape research results

Research output: Journal contributionsJournal articlesResearchpeer-review

Standard

Crowdsourcing Hypothesis Tests : Making transparent how design choices shape research results. / Landy, Justin F.; Liam, Miaolei; Ding, Isabel L. et al.

In: Psychological Bulletin, Vol. 146, No. 5, 01.05.2020, p. 451-479.

Research output: Journal contributionsJournal articlesResearchpeer-review

Harvard

Landy, JF, Liam, M, Ding, IL, Viganola, D, Tierney, W, Dreber, A, Johannesson, M, Pfeiffer, T, Ebersole, CR, Gronau, QF, Ly, A, Bergh, DVD, Marsman, M, Derks, K, Wagenmaker, EJ, Proctor, A, Bartels, DM, Bauman, CW, Brady, WJ, Cheung, F, Cimpian, A, Dohle, S, Donnellan, MB, Hahn, A, Hall, MP, Jiménez-Leal, W, Johnson, DJ, Lucas, RE, Monin, B, Montealegre, A, Mullen, E, Pang, J, Ray, J, Reinero, DA, Reynolds, J, Sowden, W, Storage, D, Su, R, Tworek, CM, Van Bavel, JJ, Walco, D, Wills, J, Xu, X, Yam, KC, Yang, X, Cunningham, WA, Schweinsberg, M, Urwitz, M, Uhlmann, EL, Adamkovic, M, Alaei, R, Albers, CJ, Allard, A, Anderson, IA, Andreychik, MR, Babinčák, P, Baker, BJ, Baník, G, Baskin, E, Bavolar, J, Berkers, RMWJ, Białek, M, Blanke, J, Breuer, J, Brizi, A, Brown, SEV, Brühlmann, F, Bruns, H, Caldwell, L, Campourcy, JF, Chan, EY, Chang, YP, Cheung, BY, Chin, A, Cho, KW, Columbus, S, Conway, P, Corretti, CA, Craig, AW, Curran, PG, Danvers, AF, Dawson, IGJ, Day, MV, Dietl, E, Doerflinger, JT, Dominici, A, Dranseika, V, Edelsbrunner, PA, Edlund, JE, Fisher, M, Fung, A, Genschow, O, Gnambs, T, Goldberg, MH, Graf-Vlachy, L, Hafenbrack, AC, Hafenbrädl, S, Hartanto, A, Heffner, JP, Hilgard, J, Holzmeister, F, Horchak, OV, Huang, TST, Hüffmeier, J, Hughes, S, Hussey, I, Imhoff, R, Jaeger, B, Jamro, K, Johnson, SGB, Jones, A, Keller, L, Kombeiz, O, Krueger, LE, Lantian, A, Laplante, JP, Lazarevic, LB, Leclerc, J, Legate, N, Leonhardt, JM, Leung, DW, Levitan, CA, Lin, H, Liu, Q, Liuzza, MT, Locke, KD, Ly, AL, MacEacheron, M, Madan, CR, Manley, H, Mari, S, Martončik, M, McLean, SL, McPhetres, J, Mercier, BG, Michels, C, Mullarkey, MC, Musser, ED, Nalborczyk, L, Nilsonne, G, Otis, NG, Otner, SMG, Otto, PE, Oviedo-Trespalacios, O, Paruzel-Czachura, M, Pellegrini, F, Pereira, VMD, Perfecto, H, Pfuhl, G, Phillips, MH, Plonsky, O, Pozzi, M, Puric, DB, Raymond-Barker, B, Redman, DE, Reynolds, CJ, Ropovik, I, Röseler, L, Ruessmann, JK, Ryan, WH, Sablaturova, N, Schuepfer, KJ, Schütz, A, Sirota, M, Stefan, M, Stocks, EL, Strosser, GL, Suchow, JW, Szabelska, A, Tey, KS, Tiokhin, L, Troian, J, Utesch, T, Vásquez-Echeverriá, A, Vaughn, LA, Verschoor, M, Helversen, BV, Wallisch, P, Weissgerber, SC, Wichman, AL, Woike, JK, Žeželj, I, Zickfeld, JH, Ahn, Y, Blaettchen, PF, Kang, X, Lee, YJ, Parker, PM, Parker, PA, Song, JS, Very, MA & Wong, L 2020, 'Crowdsourcing Hypothesis Tests: Making transparent how design choices shape research results', Psychological Bulletin, vol. 146, no. 5, pp. 451-479. https://doi.org/10.1037/bul0000220

APA

Landy, J. F., Liam, M., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., Ly, A., Bergh, D. V. D., Marsman, M., Derks, K., Wagenmaker, E. J., Proctor, A., Bartels, D. M., Bauman, C. W., Brady, W. J., ... Wong, L. (2020). Crowdsourcing Hypothesis Tests: Making transparent how design choices shape research results. Psychological Bulletin, 146(5), 451-479. https://doi.org/10.1037/bul0000220

Vancouver

Landy JF, Liam M, Ding IL, Viganola D, Tierney W, Dreber A et al. Crowdsourcing Hypothesis Tests: Making transparent how design choices shape research results. Psychological Bulletin. 2020 May 1;146(5):451-479. doi: 10.1037/bul0000220

Bibtex

@article{dd49e901233e4c37aa842c6b5099c5b0,
title = "Crowdsourcing Hypothesis Tests: Making transparent how design choices shape research results",
abstract = "To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N = 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = —0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.",
keywords = "conceptual replications, crowdsourcing, forecasting, research robustness, scientific transparency, Business psychology",
author = "Landy, {Justin F.} and Miaolei Liam and Ding, {Isabel L.} and Domenico Viganola and Warren Tierney and Anna Dreber and Magnus Johannesson and Thomas Pfeiffer and Ebersole, {Charles R.} and Gronau, {Quentin F.} and Alexander Ly and Bergh, {Don Van Den} and Maarten Marsman and Koen Derks and Wagenmaker, {Eric Jan} and Andrew Proctor and Bartels, {Daniel M.} and Bauman, {Christopher W.} and Brady, {William J.} and Felix Cheung and Andrei Cimpian and Simone Dohle and Donnellan, {M. Brent} and Adam Hahn and Hall, {Michael P.} and William Jim{\'e}nez-Leal and Johnson, {David J.} and Lucas, {Richard E.} and Beno{\^I}t Monin and Andres Montealegre and Elizabeth Mullen and Jun Pang and Jennifer Ray and Reinero, {Diego A.} and Jesse Reynolds and Walter Sowden and Daniel Storage and Runkun Su and Tworek, {Christina M.} and {Van Bavel}, {Jay J.} and Daniel Walco and Julian Wills and Xiaobing Xu and Yam, {Kai Chi} and Xiaoyu Yang and Cunningham, {William A.} and Martin Schweinsberg and Molly Urwitz and Uhlmann, {Eric L.} and Mat{\'u} Adamkovic and Ravin Alaei and Albers, {Casper J.} and Aur{\'e}lien Allard and Anderson, {Ian A.} and Andreychik, {Michael R.} and Peter Babin{\v c}{\'a}k and Baker, {Bradley J.} and Gabriel Ban{\'i}k and Ernest Baskin and Jozef Bavolar and Berkers, {Ruud M.W.J.} and Michal Bia{\l}ek and Joel Blanke and Johannes Breuer and Ambra Brizi and Brown, {Stephanie E.V.} and Florian Br{\"u}hlmann and Hendrik Bruns and Leigh Caldwell and Campourcy, {Jean Fran{\c c}ois} and Chan, {Eugene Y.} and Chang, {Yen Ping} and Cheung, {Benjamin Y.} and Alycia Chin and Cho, {Kit W.} and Simon Columbus and Paul Conway and Corretti, {Conrad A.} and Craig, {Adam W.} and Curran, {Paul G.} and Danvers, {Alexander F.} and Dawson, {Ian G.J.} and Day, {Martin V.} and Erik Dietl and Doerflinger, {Johannes T.} and Alice Dominici and Vilius Dranseika and Edelsbrunner, {Peter A.} and Edlund, {John E.} and Matthew Fisher and Anna Fung and Oliver Genschow and Timo Gnambs and Goldberg, {Matthew H.} and Lorenz Graf-Vlachy and Hafenbrack, {Andrew C.} and Sebastian Hafenbr{\"a}dl and Andree Hartanto and Heffner, {Joseph P.} and Joseph Hilgard and Felix Holzmeister and Horchak, {Oleksandr V.} and Huang, {Tina S.T.} and Joachim H{\"u}ffmeier and Sean Hughes and Ian Hussey and Roland Imhoff and Bastian Jaeger and Konrad Jamro and Johnson, {Samuel G.B.} and Andrew Jones and Lucas Keller and Olga Kombeiz and Krueger, {Lacy E.} and Anthony Lantian and Laplante, {Justin P.} and Lazarevic, {Ljiljana B.} and Jonathan Leclerc and Nicole Legate and Leonhardt, {James M.} and Leung, {Desmond W.} and Levitan, {Carmel A.} and Hause Lin and Qinglan Liu and Liuzza, {Marco Tullio} and Locke, {Kenneth D.} and Ly, {Albert L.} and Melanie MacEacheron and Madan, {Christopher R.} and Harry Manley and Silvia Mari and Marcel Marton{\v c}ik and McLean, {Scott L.} and Jonathon McPhetres and Mercier, {Brett G.} and Corinna Michels and Mullarkey, {Michael C.} and Musser, {Erica D.} and Ladislas Nalborczyk and Gustav Nilsonne and Otis, {Nicholas G.} and Otner, {Sarah M.G.} and Otto, {Philipp E.} and Oscar Oviedo-Trespalacios and Mariola Paruzel-Czachura and Francesco Pellegrini and Pereira, {Vitor M.D.} and Hannah Perfecto and Gerit Pfuhl and Phillips, {Mark H.} and Ori Plonsky and Maura Pozzi and Puric, {Danka B.} and Brett Raymond-Barker and Redman, {David E.} and Reynolds, {Caleb J.} and Ivan Ropovik and Lukas R{\"o}seler and Ruessmann, {Janna K.} and Ryan, {William H.} and Nika Sablaturova and Schuepfer, {Kurt J.} and Astrid Sch{\"u}tz and Miroslav Sirota and Matthias Stefan and Stocks, {Eric L.} and Strosser, {Garrett L.} and Suchow, {Jordan W.} and Anna Szabelska and Tey, {Kian Siong} and Leonid Tiokhin and Jais Troian and Till Utesch and Alejandro V{\'a}squez-Echeverri{\'a} and Vaughn, {Leigh Ann} and Mark Verschoor and Helversen, {Bettina Von} and Pascal Wallisch and Weissgerber, {Sophia C.} and Wichman, {Aaron L.} and Woike, {Jan K.} and Iris {\v Z}e{\v z}elj and Zickfeld, {Janis H.} and Yeonsin Ahn and Blaettchen, {Philippe F.} and Xi Kang and Lee, {Yoo Jin} and Parker, {Philip M.} and Parker, {Paul A.} and Song, {Jamie S.} and Very, {May Anne} and Lynn Wong",
note = "Publisher Copyright: {\textcopyright} 2020, American Psychological Association.",
year = "2020",
month = may,
day = "1",
doi = "10.1037/bul0000220",
language = "English",
volume = "146",
pages = "451--479",
journal = "Psychological Bulletin",
issn = "0033-2909",
publisher = "American Psychological Association Inc.",
number = "5",

}

RIS

TY - JOUR

T1 - Crowdsourcing Hypothesis Tests

T2 - Making transparent how design choices shape research results

AU - Landy, Justin F.

AU - Liam, Miaolei

AU - Ding, Isabel L.

AU - Viganola, Domenico

AU - Tierney, Warren

AU - Dreber, Anna

AU - Johannesson, Magnus

AU - Pfeiffer, Thomas

AU - Ebersole, Charles R.

AU - Gronau, Quentin F.

AU - Ly, Alexander

AU - Bergh, Don Van Den

AU - Marsman, Maarten

AU - Derks, Koen

AU - Wagenmaker, Eric Jan

AU - Proctor, Andrew

AU - Bartels, Daniel M.

AU - Bauman, Christopher W.

AU - Brady, William J.

AU - Cheung, Felix

AU - Cimpian, Andrei

AU - Dohle, Simone

AU - Donnellan, M. Brent

AU - Hahn, Adam

AU - Hall, Michael P.

AU - Jiménez-Leal, William

AU - Johnson, David J.

AU - Lucas, Richard E.

AU - Monin, BenoÎt

AU - Montealegre, Andres

AU - Mullen, Elizabeth

AU - Pang, Jun

AU - Ray, Jennifer

AU - Reinero, Diego A.

AU - Reynolds, Jesse

AU - Sowden, Walter

AU - Storage, Daniel

AU - Su, Runkun

AU - Tworek, Christina M.

AU - Van Bavel, Jay J.

AU - Walco, Daniel

AU - Wills, Julian

AU - Xu, Xiaobing

AU - Yam, Kai Chi

AU - Yang, Xiaoyu

AU - Cunningham, William A.

AU - Schweinsberg, Martin

AU - Urwitz, Molly

AU - Uhlmann, Eric L.

AU - Adamkovic, Matú

AU - Alaei, Ravin

AU - Albers, Casper J.

AU - Allard, Aurélien

AU - Anderson, Ian A.

AU - Andreychik, Michael R.

AU - Babinčák, Peter

AU - Baker, Bradley J.

AU - Baník, Gabriel

AU - Baskin, Ernest

AU - Bavolar, Jozef

AU - Berkers, Ruud M.W.J.

AU - Białek, Michal

AU - Blanke, Joel

AU - Breuer, Johannes

AU - Brizi, Ambra

AU - Brown, Stephanie E.V.

AU - Brühlmann, Florian

AU - Bruns, Hendrik

AU - Caldwell, Leigh

AU - Campourcy, Jean François

AU - Chan, Eugene Y.

AU - Chang, Yen Ping

AU - Cheung, Benjamin Y.

AU - Chin, Alycia

AU - Cho, Kit W.

AU - Columbus, Simon

AU - Conway, Paul

AU - Corretti, Conrad A.

AU - Craig, Adam W.

AU - Curran, Paul G.

AU - Danvers, Alexander F.

AU - Dawson, Ian G.J.

AU - Day, Martin V.

AU - Dietl, Erik

AU - Doerflinger, Johannes T.

AU - Dominici, Alice

AU - Dranseika, Vilius

AU - Edelsbrunner, Peter A.

AU - Edlund, John E.

AU - Fisher, Matthew

AU - Fung, Anna

AU - Genschow, Oliver

AU - Gnambs, Timo

AU - Goldberg, Matthew H.

AU - Graf-Vlachy, Lorenz

AU - Hafenbrack, Andrew C.

AU - Hafenbrädl, Sebastian

AU - Hartanto, Andree

AU - Heffner, Joseph P.

AU - Hilgard, Joseph

AU - Holzmeister, Felix

AU - Horchak, Oleksandr V.

AU - Huang, Tina S.T.

AU - Hüffmeier, Joachim

AU - Hughes, Sean

AU - Hussey, Ian

AU - Imhoff, Roland

AU - Jaeger, Bastian

AU - Jamro, Konrad

AU - Johnson, Samuel G.B.

AU - Jones, Andrew

AU - Keller, Lucas

AU - Kombeiz, Olga

AU - Krueger, Lacy E.

AU - Lantian, Anthony

AU - Laplante, Justin P.

AU - Lazarevic, Ljiljana B.

AU - Leclerc, Jonathan

AU - Legate, Nicole

AU - Leonhardt, James M.

AU - Leung, Desmond W.

AU - Levitan, Carmel A.

AU - Lin, Hause

AU - Liu, Qinglan

AU - Liuzza, Marco Tullio

AU - Locke, Kenneth D.

AU - Ly, Albert L.

AU - MacEacheron, Melanie

AU - Madan, Christopher R.

AU - Manley, Harry

AU - Mari, Silvia

AU - Martončik, Marcel

AU - McLean, Scott L.

AU - McPhetres, Jonathon

AU - Mercier, Brett G.

AU - Michels, Corinna

AU - Mullarkey, Michael C.

AU - Musser, Erica D.

AU - Nalborczyk, Ladislas

AU - Nilsonne, Gustav

AU - Otis, Nicholas G.

AU - Otner, Sarah M.G.

AU - Otto, Philipp E.

AU - Oviedo-Trespalacios, Oscar

AU - Paruzel-Czachura, Mariola

AU - Pellegrini, Francesco

AU - Pereira, Vitor M.D.

AU - Perfecto, Hannah

AU - Pfuhl, Gerit

AU - Phillips, Mark H.

AU - Plonsky, Ori

AU - Pozzi, Maura

AU - Puric, Danka B.

AU - Raymond-Barker, Brett

AU - Redman, David E.

AU - Reynolds, Caleb J.

AU - Ropovik, Ivan

AU - Röseler, Lukas

AU - Ruessmann, Janna K.

AU - Ryan, William H.

AU - Sablaturova, Nika

AU - Schuepfer, Kurt J.

AU - Schütz, Astrid

AU - Sirota, Miroslav

AU - Stefan, Matthias

AU - Stocks, Eric L.

AU - Strosser, Garrett L.

AU - Suchow, Jordan W.

AU - Szabelska, Anna

AU - Tey, Kian Siong

AU - Tiokhin, Leonid

AU - Troian, Jais

AU - Utesch, Till

AU - Vásquez-Echeverriá, Alejandro

AU - Vaughn, Leigh Ann

AU - Verschoor, Mark

AU - Helversen, Bettina Von

AU - Wallisch, Pascal

AU - Weissgerber, Sophia C.

AU - Wichman, Aaron L.

AU - Woike, Jan K.

AU - Žeželj, Iris

AU - Zickfeld, Janis H.

AU - Ahn, Yeonsin

AU - Blaettchen, Philippe F.

AU - Kang, Xi

AU - Lee, Yoo Jin

AU - Parker, Philip M.

AU - Parker, Paul A.

AU - Song, Jamie S.

AU - Very, May Anne

AU - Wong, Lynn

N1 - Publisher Copyright: © 2020, American Psychological Association.

PY - 2020/5/1

Y1 - 2020/5/1

N2 - To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N = 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = —0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.

AB - To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N = 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = —0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.

KW - conceptual replications

KW - crowdsourcing

KW - forecasting

KW - research robustness

KW - scientific transparency

KW - Business psychology

UR - http://www.scopus.com/inward/record.url?scp=85081412411&partnerID=8YFLogxK

U2 - 10.1037/bul0000220

DO - 10.1037/bul0000220

M3 - Journal articles

C2 - 31944796

AN - SCOPUS:85081412411

VL - 146

SP - 451

EP - 479

JO - Psychological Bulletin

JF - Psychological Bulletin

SN - 0033-2909

IS - 5

ER -

DOI