The vast majority of experimental work in behavioral and biomedical science involves group comparison—in the simplest case, an experimental and a control group. Average data are compared and the variability within each group is used to estimate the probability that any mean difference could have occurred by chance. The estimation method typically used, the Null Hypothesis Statistical Test (NHST) method, was devised by Ronald Fisher in a context to be discussed in a moment.

There are serious problems going from group data to the properties of individuals (the object of inquiry problem: Staddon, 2019), but the first major problem encountered by the NHST method was replication, exposed in a landmark paper by John Ioannidis (2005). The replication problem is fixable and various solutions have been proposed. In July 2017 a letter to Science (Benjamin et al.), signed by more than 70 statisticians, suggested that a solution to the NHST-replicability problem is to set the criterion (alpha level) for rejecting the null (no-effect) hypothesis at p = .005 rather than Ronald Fisher’s suggestion of p = .05, the then-standard. The authors argued that rather than choosing a one in twenty (or less) chance of error as good enough to accept the hypothesis that your treatment has an effect, the standard should be upped to one in two hundred.

Replicability would sometimes be improved by a tougher criterion; but a p-value this small would also eliminate much social science research that uses NHST; the publication rate in social and biomedical science would plummet. Partly for this reason, more than 80 scientists signed a November 2017 letter (Lakens et al, 2017) to Nature rejecting the suggestion of Benjamin et al., instead recommending “that the label ‘statistically significant’ should no longer be used” and concluding instead “that researchers should transparently report and justify all choices they make when designing a study, including the alpha [critical p-value] level.”