Comments on: Make up your own rules of probability

By: Reproducible research | Jarad Niemi

Reproducible research | Jarad Niemi — Mon, 19 Sep 2011 17:42:15 +0000

[...] A new paper in Annals of Applied Statistics by Keith Baggerly and Kevin Coombes discusses Forensic bioinformatics. I heard Keith talk about this at MCMSki3. Basically, the Baggerly and Coombes looked at a few bioinformatics papers, tried to reproduce some statistical analyses, and failed. In failing, Baggerly and Coombes realized the authors of those papers had not done what they said they had done. So Baggerly and Coombes reverse engineered what was actually done with astounding results, e.g. training data with sensitive/resistant labels reversed and bizarre probability laws [P(A,B) = max(P(A),P(B)). [...]

By: Reproducibility in Practice « Statistical Modeling, Causal Inference, and Social Science

Thu, 15 Sep 2011 17:03:26 +0000

[...] based on simple visual inspection and counting, and are not documented further.”There are also gross statistical errors.Continuing in the usual theme of my occasional posts, I’ll share what reproducible research [...]

By: Joey

Joey — Tue, 09 Aug 2011 20:21:27 +0000

The RHS of the second rule, max( P(A), P(B) ), jumps out as what is the standard rule for disjunction (the T-conorm) in fuzzy logic. And it’s not unusual in that domain to be sloppy and just call everything a probability even if you’re considering something less constrained. So, one could be more charitable and suppose that the authors had something like this in mind, but even so, that’s probably still a bit too much mind-reading.

By: John

John — Tue, 15 Feb 2011 14:08:08 +0000

Annie: I completely agree. See Three reasons to distrust microarray results.

By: Annie Pettit

Annie Pettit — Tue, 15 Feb 2011 13:20:11 +0000

Let’s not forget all the errors that are simply misapplications of statistics, for example the application of probability statistics to nonprobability samples. It’s the wild west out there.

By: The Man

The Man — Wed, 02 Feb 2011 16:04:43 +0000

Now if only the AGW scientists would release the models and data used to ‘prove’ man made global warming, we would have a treasure trove of new statistical methods…

By: Lane

Lane — Wed, 02 Feb 2011 14:57:20 +0000

Statistically speaking, we all make mistakes.
…but if the statistics are wrong, am I perfect?

By: Stan Young

Stan Young — Tue, 01 Feb 2011 16:45:14 +0000

My research indicates that most claims coming from medical observational studies that have been tested in randomized clinical trials fail to replicate. Some of the problems are covered in a lecture which can be found here:
http://niss.org/sites/default/files/Young_Safety_June_2008.pdf

By: Dan P

Dan P — Mon, 31 Jan 2011 18:35:18 +0000

And all those papers that assume some variation of P(A|B)=P(B|A), turning significance levels into the probability of a hypothesis being true.

By: Konrad

Konrad — Mon, 31 Jan 2011 17:28:25 +0000

This is a well-known problem, and it’s not restricted to statistics (nor, would I assume, to bioinformatics papers). Just pick any paper that uses differential equations to describe a biochemical system. Chances are that the numerical analysis is either completely wrong or at least incomplete. In fact, the functions often describe variations of quantities of chemicals in the cells – and you should be very wary when these functions suddenly become negative. Or take papers with analyses of gene regulatory network.

By: Trust no-one: errors and irreproducibility in public data | What You’re Doing Is Rather Desperate

Thu, 09 Sep 2010 04:57:52 +0000

[...] It escaped my attention last year, in part because “Annals of Applied Statistics” is not high on my journal radar. However, other bloggers did pick it up: see posts at Reproducible Research Ideas and The Endeavour. [...]

By: New paper in forensic bioinformatics « Reproducible Research Ideas

New paper in forensic bioinformatics « Reproducible Research Ideas — Tue, 13 Oct 2009 14:38:32 +0000

[...] blog post Make up your own rules of probability discusses a couple of the innovative rules of probability Baggerly and Coombes discovered while [...]

By: EastwoodDC

EastwoodDC — Mon, 21 Sep 2009 12:53:53 +0000

… but … but … the formulas are in the spreadsheet, so they must be right!

Sheesh.

By: David Smith

David Smith — Fri, 18 Sep 2009 23:23:44 +0000

Thanks for pointing this article out, John. It's in equal parts fascinating and terrifying. I saw Keith Baggerly talk about this at BioConductor -- it's a fascinating detective story how they pieced together the original data and figured out the response. But the lack of access to that data, and the refusal of the journal to correct the record, is scary.

By: Ken

Ken — Fri, 18 Sep 2009 22:50:38 +0000

One of the worries is that even if the stats is internally correct, they have thought some other factors weren’t important enough to tell you that affect the reliability of the results. For example how they changed their models five times before getting the analysis they wanted. This is something that is already a major problem in epidemiology, and bioinformatics just multiplies the problem. Which of a hundred food groups causes cancer becomes which of thousands of genes causes cancer.

Went to a talk at the International Biometrics a few years ago where someone (Mark Segal ?) pulled apart a bioinformatics paper by comparing what the authors did to what they put in the paper and how it totally destroyed their significance calculations.

By: John S.

John S. — Fri, 18 Sep 2009 18:55:50 +0000

I have a paper in the review process right now that points out an error in a previous paper. I put the author of that paper down as a suggested reviewer. Maybe this is why the reviews are taking so long.

By: Reproducible Ideas » Blog Archive » New paper in forensic bioinformatics

Fri, 18 Sep 2009 18:14:05 +0000

[...] blog post Make up your own rules of probability discusses a couple of the innovative rules of probability Baggerly and Coombes discovered while [...]