Comments on: Five criticisms of significance testing

By: Jeffrey Horn

Jeffrey Horn — Fri, 11 Nov 2011 23:42:55 +0000

John: Thanks for your reply.

I see the point about the interplay between sample size, effect size, and statistical significance. The way I was taught frequentist methods is to assume a null that is unlikely, and choose an alternate hypothesis that is mutually exclusive and exhaustive with the null. For example, if I’m testing whether or not adding piece-rates improves worker productivity, my null is that there is no difference in mean productivity between the treatment and control.

If I find a statistically significant difference, that tells me that a difference exists. It isn’t clear that the null is necessarily impossible, though I agree it is highly improbable. That means the test can only tell me that the incentive changed productivity, not in which direction.

I rely on the confidence interval to tell me that, and to tell me precisely how far away from zero the effect could be (under classical assumptions) in both best and worse cases.

I’m just having a hard time figuring out why the impossibility (improbability) of the null should matter for the test. I’m far more convinced of the other critiques. I’m also willing to believe authors regularly state more than the test allows them to, but it’s the job of the reader, ultimately, to figure out if they’re being snow-jobbed.

By: John

John — Fri, 11 Nov 2011 00:42:48 +0000

Jeffrey: I take Gelman’s critique to refer specifically to null hypotheses of equality, not null hypotheses in general.

People commonly assume that strong evidence against exact equality of two values is equivalent to pretty good evidence of an important gap between the two values. This isn’t necessarily so. There’s some reason for this belief: It’s easier to show statistical significance when the effect size is larger. But it’s also easier to show significance when you’ve got a lot of data. And to point #3, statistical significance is not the same as scientific significance.

By: Jeffrey Horn

Jeffrey Horn — Thu, 10 Nov 2011 22:30:46 +0000

John, how do you read the Gelman critique? I'm not sure I understand it. I was just reading an article from Brad DeLong and Kevin Lang, "Are all economic hypotheses false?", and he points out that

Although the falsificationist view of scientific methodology (Popper 1959) stresses the importance of specifying test events that cannot occur under the null hypothesis...

I haven't read Popper, so I'm wondering about this clause. Does this conflict with Gelman's critique?

By: John

John — Sun, 03 Apr 2011 19:15:06 +0000

Gelman has often said, as recently as yesterday, that testing a point-null hypothesis via Bayesian techniques is as bad as testing one via frequentist techniques.

By: Deborah Mayo

Deborah Mayo — Sun, 03 Apr 2011 19:08:28 +0000

It is disappointing (though not surprising) to see these “criticisms” from years ago without any reply, even though numerous replies exist in the literature. Just a word on the main alleged problems:
Andrew Gelman: If null hypotheses are presumed false, then why do Bayesian hypotheses tests assign a spiked degree of belief to them? They do so in order to get a disagreement with p-values, but once again, this just shows what’s wrong with the Bayesian hypothesis tests that assign spiked priors to the null (I’m not saying Gelman endorses them).
Jim Berger: This is not what a small p-value means, and it would be a fallacious use of tests to take a small p-value as evidence FOR an alternative that was just as unlikely. The inference to the alternative would have passed a highly INsevere test. There is no argument for claiming that “comparisons of hypotheses should be conditional on the data”, but plenty of evidence that this robs on of calculating any error probabilities. Now that Berger has himself abandoned the likelihood principle, apparently he agrees .
To Stephen Ziliak and Deirdra McCloskey: These authors join others in the ill-founded hysteria about the secret “cults” of significance testers that are allegedly undermining science. Any even one-quarter way respectable treatment of statistical significance tests of the frequentist variety blatantly announce that statistical significance is not the same as scientific significance. On the business of “effect size”, ways to interpret statistical tests so that the inferences are clearly in terms of the discrepancies for which there is an isn’t evidence are many. These authors do not read such defenders, since they attack D. Mayo on these grounds, even though she has developed a clear way of reporting on discrepancies (that is suprioer, by the way, to things like confidence intervals).
To John Ioannidis: It is well known that small p-values do not mean small probability of being wrong, but this fails utterly to show that we should instead be using posteriors (regardless of how they are interpreted). He needs to tell us where he gets the prior that leads to the 74% posterior—I guarantee it involves assigning a probability of p to a hypothesis because it has been randomly selected from an “urn” of hypotheses, p% of which are believed to be true! This is a fallacious and highly irrelevant calculation that leads to absurd results. For references, see D. Mayo.

By: Tweets that mention Five criticisms of significance testing — The Endeavour -- Topsy.com

Mon, 17 Jan 2011 18:49:34 +0000

[...] This post was mentioned on Twitter by Matt Gershoff, Michael Hoffman. Michael Hoffman said: @nparmalee http://www.johndcook.com/blog/2008/11/18/five-criticisms-of-significance-testing/ [...]

By: zyxo

zyxo — Mon, 17 Jan 2011 17:13:40 +0000

@Stephen & Deirdra :
Right ! Not only scientific significance but also economic significance is something else than statistic significance, and much more important.
In commercial datamining, we work with millions of customers. Almost any effect, no matter how tiny, turns out to be statistically significant.
We don’t care. The model must increase the expected surplus sales value, i.e. the economic significance must be worthwhile.

By: Dan

Dan — Fri, 21 Nov 2008 05:35:09 +0000

Significance testing tends to be over utilized. Everyone wants that p-value for their publication (and the journals demand it), so a lot of hypothesis tests are forced when a simple estimate might have been more appropriate. It important to remember that a significance test is not the end of the investigation, just a step in the process.

By: John

John — Wed, 19 Nov 2008 04:02:52 +0000

Using Bayes factors — the ratio of the posterior probabilities of the two hypotheses given the data — gets around some of the weaknesses of significance testing, especially the criticisms of Jim Berger and John Ioannidis. In fact, part of the problem with the naive use of p-values is that people often think that they are Bayes factors, without using that term. Bayes factors have an intuitive interpretation in terms of decibels by analogy to sound intensity.

By: Pedro

Pedro — Wed, 19 Nov 2008 03:50:43 +0000

I’ve never understood the alternative though. Confidence intervals? What if the best prediction a theory can make is: A and B should be different. In my field (psycholinguistics), we’re not at the point yet where we can say, under condition A, I expect at 250 ms reaction time, and under B a 300 ms reaction time.

I’m happy to accept that significance testing is a bad tool. What is a good replacement?