Comments on: The cult of significance testing

By: John Venier

John Venier — Fri, 05 Dec 2008 19:13:42 +0000

It seems that a lot of lucrative drugs are found to have ‘better’ replacements just as their patents are expiring. The manufacturer holds the patent on the replacement, naturally. Often these replacements are metabolites of the original drug which are found to have the same action as the original drug. I suspect that the studies which demonstrate a (statistically) significant increase in effectiveness are extremely overpowered and that the clinical significance is nil.

Another ‘improvement’ strategy is simply using the more active stereoisomer (enantiomer) exclusively instead of both. The non-racemic drug can be patented, named, and marketed as a new drug, resulting in literally billions of dollars of profit. Prilosec and Nexium are a great example of this. I would imagine that with billions of dollars at stake AstraZenica tried every way imaginable to demonstrate any statistically significant improvement, however meager. From what I understand (and I am no specialist) there is actually no clinically meaningful difference between the drugs, but Nexium has a shiny new patent and Prilosec is now available as a generic OTC.

By: John Johnson

John Johnson — Fri, 05 Dec 2008 18:18:28 +0000

What amazes me is all the Phase 2 studies that “fail to meet endpoint” or “miss statistical significance” when those studies aren’t even powered in the first place. Or, even worse, the practice of “p-value fishing,” i.e. the attempt to cover up a worthless drug by showing a p-value less than 0.05.

This “cult of statistical significance” has led to many bizarre behaviors and habits in the scientific community, ones I think we would do well to expunge.

By: ekzept

ekzept — Thu, 04 Dec 2008 04:31:19 +0000

There is a heartening discussion of this in Kraemer and Thiemann, How Many Subjects? Statistical Power Analysis In Research. The other context it is dealt with is in the careful comparison of Bayesian vs Frequentist decision theory, per J.O.Berger's treatment, Statistical Decision Theory and Bayesian Analysis which I found thorough but difficult. Finally, a recent one: D.J.Murdoch, Y.-L. Tsai, J.Adcock, "P-values are random variables", The American Statistician, 62(3), 242-245.

By: John Venier

John Venier — Wed, 03 Dec 2008 18:28:42 +0000

Amen to that. Some examples come to mind — Linus Pauling’s testing of the benefits of vitamin C showed statistical significance which was clinically trivial — his study had a large sample size.

Every time you see an ad on TV which says something like “no toothpaste was shown to be better than ours” mentally add “in our test with no power”.

One professor I know did some consulting for an unnamed hospital which was being sued by patients who contacted infections after hip replacement surgery. The professor was hired as an expert witness to testify whether the frequency of such infections at that hospital was worse than expected or not. When he analzyed the data as a whole there was no significant evidence that their rate was higher than expected. But the hospital had three surgical teams which performed those surgeries. When the data were grouped according to teams, one team was clearly much more likely to leave patients with an infection. As it turned out, that team was relatively new compared to the other two, and evidently it makes a big difference how long a team has been working together.

I’ve seen other cutoffs in other fields — 0.10 and 0.20 in fields such as psychology, sociology, and biology. But the papers I saw still used cutoffs to assess significance. I recall that they did publish the actual p-values that they obtained.

By: Scott

Scott — Wed, 03 Dec 2008 15:35:03 +0000

I could not agree more with this, John. It amazes me that everyone “knows” that effect size is important and the .05 cutoff is arbitrary, but published research continues to center around significance anyway. As you know, not only does decreasing the sample size remove significance, even the tiniest effect becomes significant if your sample size is large enough. In the field of education this means that almost anything is significant when thousands of children are tested. (Just about anything in a classroom has at least a tiny effect on just about anything else.) I am currently distressed over certain questionable practices that are being pushed because research has shown their “significance” even though effect sizes are clearly very small. Unfortunately the currently published research on these practices is not talking about the effect sizes at all.

John, how can academia’s knowledge of statistics be so sophisticated and their research be so ignorant of these basics?