Comments on: Five criticisms of significance testing http://www.johndcook.com/blog/2008/11/18/five-criticisms-of-significance-testing/ The blog of John D. Cook Sat, 31 Jul 2010 07:45:27 -0400 http://wordpress.org/?v=2.8.4 hourly 1 By: Dan http://www.johndcook.com/blog/2008/11/18/five-criticisms-of-significance-testing/comment-page-1/#comment-10028 Dan Fri, 21 Nov 2008 05:35:09 +0000 http://www.johndcook.com/blog/?p=902#comment-10028 Significance testing tends to be over utilized. Everyone wants that p-value for their publication (and the journals demand it), so a lot of hypothesis tests are forced when a simple estimate might have been more appropriate. It important to remember that a significance test is not the end of the investigation, just a step in the process. Significance testing tends to be over utilized. Everyone wants that p-value for their publication (and the journals demand it), so a lot of hypothesis tests are forced when a simple estimate might have been more appropriate. It important to remember that a significance test is not the end of the investigation, just a step in the process.

]]> By: John http://www.johndcook.com/blog/2008/11/18/five-criticisms-of-significance-testing/comment-page-1/#comment-9928 John Wed, 19 Nov 2008 04:02:52 +0000 http://www.johndcook.com/blog/?p=902#comment-9928 Using Bayes factors — the ratio of the posterior probabilities of the two hypotheses given the data — gets around some of the weaknesses of significance testing, especially the criticisms of Jim Berger and John Ioannidis. In fact, part of the problem with the naive use of p-values is that people often think that they <i>are</i> Bayes factors, without using that term. Bayes factors have an intuitive interpretation in terms of <a href="http://www.johndcook.com/blog/2008/03/11/how-loud-is-the-evidence/" rel="nofollow">decibels</a> by analogy to sound intensity. Using Bayes factors — the ratio of the posterior probabilities of the two hypotheses given the data — gets around some of the weaknesses of significance testing, especially the criticisms of Jim Berger and John Ioannidis. In fact, part of the problem with the naive use of p-values is that people often think that they are Bayes factors, without using that term.

Bayes factors have an intuitive interpretation in terms of decibels by analogy to sound intensity.

]]> By: Pedro http://www.johndcook.com/blog/2008/11/18/five-criticisms-of-significance-testing/comment-page-1/#comment-9927 Pedro Wed, 19 Nov 2008 03:50:43 +0000 http://www.johndcook.com/blog/?p=902#comment-9927 I've never understood the alternative though. Confidence intervals? What if the best prediction a theory can make is: A and B should be different. In my field (psycholinguistics), we're not at the point yet where we can say, under condition A, I expect at 250 ms reaction time, and under B a 300 ms reaction time. I'm happy to accept that significance testing is a bad tool. What is a good replacement? I’ve never understood the alternative though. Confidence intervals? What if the best prediction a theory can make is: A and B should be different. In my field (psycholinguistics), we’re not at the point yet where we can say, under condition A, I expect at 250 ms reaction time, and under B a 300 ms reaction time.

I’m happy to accept that significance testing is a bad tool. What is a good replacement?

]]>