Replication crisis

The replication crisis in science refers to the persistent failure of scientific studies, across a range of fields, to replicate at the nominal rate of the statistical hypothesis tests and confidence intervals.

Literature

Biology and biomedicine

  • Ioannidis, 2005: Why most published research findings are false (doi, pdf)
    • The famous, or infamous, paper by John Ioannidis

Social sciences, especially psychology

  • Simmons, Nelson, Simonsohn, 2011: False-positive psychology (doi, pdf)
    • On “research degrees of freedom” in psychology research
  • Gelman & Loken, 2013: The garden of forking paths (pdf)
    • See also: Gelman & Loken, 2014: The statistical crisis in science (doi, pdf)
    • On “garden of forking paths” as unconscious form of p-hacking
    • From this point of view, “mining the data” (p. 8) is bad and should be avoided
  • Gelman et al, 2016: Increasing transparency through a multiverse analysis (doi, pdf, Gelman’s blog )
    • On explicitly constructing each path through the garden to form a “multiverse”, a form of sensitivity analysis
    • Focused on data multiverse (data preparation), not model multiverse (p. 18)
    • No explicitly statistical advice beyond averaging the p-values in the multiverse: “This mean value can be considered as the p-value of a hypothetical pre-registered study with conditions chosen at random among the possibilities in the multiverse and seems like a fair measurement in a setting where all of the possible data processing choices seem plausible” (p. 18)
  • Leek et al, 2017: Five ways to fix statistics (doi, pdf, Gelman’s blog )
    • Contributions by Leek, Gelman, Colquhoun, Nuijten, and Goodman
    • Leeks argue that we should study human cognition and behavior re: data analysis
  • Yarkoni, 2019: The generalizability crisis (psyarxiv , Gelman’s blog )