Scientific results fading over time

A recent article in The New Yorker gives numerous examples of scientific results fading over time. Effects that were large when first measured become smaller in subsequent studies. Firmly established facts become doubtful. It’s as if scientific laws are being gradually repealed. This phenomena is known as “the decline effect.” The full title of the article is The decline effect and the scientific method.

The article brings together many topics that have been discussed here: regression to the mean, publication bias, scientific fashion, etc. Here’s a little sample.

“… when I submitted these null results I had difficulty getting them published. The journals only wanted confirming data. It was too exciting an idea to disprove, at least back then.” … After a new paradigm is proposed, the peer-review process is tilted toward positive results. But then, after a few years, the academic incentives shift—the paradigm has become entrenched—so that the most notable results are now those that disprove the theory.

This excerpt happens to be talking about “fluctuating asymmetry,” the idea that animals prefer more symmetric mates because symmetry is a proxy for good genes. (I edited out references to fluctuating asymmetry from the quote to emphasize that the remarks could equally apply to any number of topics. ) Fluctuating asymmetry was initially confirmed by numerous studies, but then the tide shifted and more studies failed to find the effect.

When such a shift happens, it would be reassuring to believe that the initial studies were simply wrong and that the new studies are right. But both the positive and negative results confirmed the prevailing view at the time they were published. There’s no reason to believe the latter studies are necessarily more reliable.

Related posts:

6 thoughts on “Scientific results fading over time

  1. Here’s an idea: the Journal of Experimental Designs

    Before submission, submitters do a complete experiment. Then they send a half-article where only the design of the experiment and statistical methods used are described. Publication is decided solely on the basis of this description. After the decision whether to publish has been taken, the results are revealed.

    One might want to move the actual realization of the experiment after the decision to publish, but that would cause too many logistics problems.

    There are issues that this wouldn’t solve, but focusing the evaluation on the design of the experiment rather than on the results would steer both submitters and reviewers away from some of the dangers that Richard Feynman was already warning about in his famous speech in 1974. http://calteches.library.caltech.edu/51/2/CargoCult.pdf

  2. We have to learn to embrace new ideas without getting caught in the hype.

    @Cuoq

    PLoS works largely how you describe. They only check for correctedness. And PLoS journals seem to do very well.

  3. This seems like a terrible case of observation bias. Of course we discover new things all the time, and old ideas are replaced or updated. That ignores everything we do know that isn’t declining or changing.

    We need to accept that some of what we know is (to some degree) wrong. The alternative is to assume all we know is right, and reject all new knowledge. That would be a real decline.

  4. I read the article, and, please correct me if I’m wrong, but the author says The most likely explanation for the decline is an obvious one: regression to the mean., he’s confused, no? There’s no linear best-fit between multiple predictors and one prediction going on here.

  5. He doesn’t mean “regression” as in linear regression. He’s using the term in it’s original sense. When you see an extreme observation, it may be partially due to underlying causes and partially due to other factors. Those other factors may not happen again the next time, and so the next observation tends to be closer to the average, i.e. has regressed toward the mean. So, for example, if an athlete does really well in one game, he’s likely to not do so well in the next one because part of the reason he did well the first time was luck that probably won’t happen again.

    I believe the term “regression to the mean” goes back to Galton in the context of height studies.

Leave a Reply

Your email address will not be published. Required fields are marked *