Closet Bayesian

When I was a grad student, a statistics postdoc confided to me that he was a “closet Bayesian.” This sounded absolutely bizarre. Why would someone be secretive about his preferred approach to statistics? I could not imagine someone whispering that although she’s doing her thesis in algebra, she’s secretively interested in differential equations.

I knew nothing about statistics at the time and was surprised to find that there was a bitter rivalry between two schools of statistics. The rivalry is still there, though it’s not as bitter as it once was.

I find it grating when someone asks “Are you a Bayesian?” It implies an inappropriate degree of commitment and exclusivity. Bayesian statistics is just a tool. Statistics itself is just tool, one way of understanding the world.

My car has a manual transmission. I prefer manual transmissions. But if someone asked whether I was a manual transmissionist, I’d look at them like they’re crazy. I don’t have any moral objections to automatic transmissions.

I evaluate a car by how well it works. And for most purposes, I prefer the way a manual transmission works. But when I’m teaching one of my kids to drive, we go out in my wife’s car with an automatic transmission. Similarly, I evaluate a mathematical model (statistical or otherwise) by how it works for a given purpose. Sometimes a Bayesian and a frequentist approach lead to the same conclusions, but the latter is easier to understand or implement. Sometimes a Bayesian method leads to a better result because it can use more information or is easier to interpret. Sometimes it’s a toss up and I use a Bayesian approach because its more familiar, just like my old car.

Related post: Bayes isn’t magic

22 thoughts on “Closet Bayesian

  1. John, can you give a few examples where the non-Bayesian approach is easier to understand? I’d love to highlight them when I’m teaching Bayesian stats.

  2. Eric: Sometimes a frequentist estimator is very simple, requiring pencil and paper rather than MCMC simulation. Such an estimator is easier to understand, or rather gives a greater feeling of understanding. Now as to what the estimator means, that’s another matter. :)

  3. From the non-statistician viewpoint, help me understand ‘knowledge-free’ prior estimation and I’ll happily roll- well, provided it doesn’t involve explaining it to reviewers every time. The elegance of Bayes’ is undermined by the (at least to a non-specialist) secret-sauce nature of these techniques- often it feels like all the weird assumptions are front-loaded into the prior process, rather than secreted in the test as in frequentist methods, where the Ho/Ha setup elegantly captures the general opinion that differences are probably interesting.

  4. Craig: All statistical inference involves subjective assumptions; knowledge-free inference is impossible. But frequentist statistics does do a better job of hiding its subjective assumptions.

    There are many approaches to overcoming the objection to prior distributions. Here are a few off the top of my head.

    1. Do a sensitivity analysis and show that conclusions are not sensitive to priors (if that’s true).
    2. Special case of sensitivity analysis: show that skeptical and optimistic priors lead to the same conclusion.
    3. Use “objective” priors, i.e. priors chosen to satisfy some mathematical optimality condition rather than prior belief.
    4. Use empirical Bayes.
    5. Treat the Bayesian procedure as a black box and study its frequentist operating characteristics.
  5. Perhaps you are a closet manual-transmissionist. I am now a closet automatic-transmissionist.

    Happy New Year, GW

  6. I am a Bayesian hypocrite. The Bayesian approach is the philosophically sounder and more precise approach. You should always use it, as long as conjugate priors work. :)

  7. Tomas: Maybe you should call yourself a pragmatic Bayesian. I agree that Bayesian statistics has a better philosophical foundation. But you then have to ask whether a Bayesian approach in practice is better than a frequentist approach in practice for your particular problem. The philosophical argument for Bayesian analysis doesn’t take into account approximately specified priors, availability of quality software, how well a model is understood, time required to do an analysis, sociological and legal constraints, etc.

  8. I like the tool metaphor. I’ve been asked “are you an OO programmer”. I usually respond “I’m a programmer and OO is in my toolbox”. I’m a programming paradigm omnivore. Perhaps you are statistically promiscuous…er, you know what I mean ;-)

    For more on Bayes see:

    The Theory That Would Not Die: How Bayes’ Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy – Sharon Bertsch McGrayne

    An Introduction to Probability and Inductive Logic – Ian Hacking

  9. John (re: Tomas) or just how freaking hard it can be to convince reviewers that your newfangled methods are right. I always worry if I’m going to be spending “reviewer political capital” arguing for methods that I like and they may hate/not-understand.

  10. Eric: I understand. There can be tremendous political incentives to stick with traditional methods. Scientists and bureaucrats who don’t understand statistics are extremely conservative about statistical methods. They think what ever they learned is the “right” thing to do and have no to way to evaluate anything new.

    I’ve also seen pressure the other way, to use Bayesian methods just because they’re sexy. Some people go to tremendous effort — or ask other people to go to tremendous effort on their behalf– to tune a Bayesian method to behave like a comparable frequentist method. Why not just use the frequentist method they’re trying to ape? Because they get brownie points for using Bayesian methods.

  11. I’m almost finished reading Nate Silver’s book, ” The Signal and the Noise: Why Most Predictions Fail – But Some Don’t” and he discusses Bayesian methods quite a bit, from the perspective of making predictions. As a lapsed mathematician (I’ve been a tech writer for the last 20+ years), I found it quite accessible and it made a lot of sense to me.

  12. The Nate’s book is great, but the Bayesian discussion on the book is quite bizarre. Some of the claims there doesn’t make sense. And I say that as a “Bayesian” myself (actually I agree with John on being a Bayesian). In any case, Nate Silver made claims like: if Fisher didn’t opposed Bayesian statistics, he’d accept that smoking causes cancer. Not only he doesn’t present any evidence in favor of this, but it doesn’t make sense. There is nothing special in Bayesian methods to allow you to uncover causality.

  13. You might say you are neither Frequenist nor Bayesian, you are an Opportunist, which according to Bruce Lee is best (in a slightly different context..).
    E

  14. John,
    I recently heard someone describe himself as a ‘maximum likelihood guy’. He said that the bayesian-frequentest thing was a false choice and that maximum likelihood methods should be given their own category. Any thoughts?

  15. Peter: There are people who are likelihoodists. They’re a pretty small group and tend to align with Bayesians. I don’t know much about that school of thought. I’ve only met one person who was enthusiastic for it.

    Likelihood inference is intuitively appealing and easy to understand. But here are a couple criticisms.

    You could think of likelihood inference as Bayesian inference with uniform (improper) priors. A Bayesian might object that this puts too much prior weight on parameter values known to be unlikely or impossible. Another criticism would be that posterior mode (which is what maximum likelihood is, if you use a uniform prior) can be less robust than posterior mean.

  16. Interesting. Completely new to me. I finally need to get a bit more into statistics.

    Funny thing is that I would (contrary to other posters here) say that I intuitively find the Bayesian approach not that well founded philosophically (stemming from what little I know about epistemology). But I am not too sure about the Frequentist approach either. The Bayesian approach seems more intuitive practically, though.

  17. If you haven’t seen the E.T. Jaynes book, it makes an awesome read. Basically argues that Bayes is the extension of logic into the realm in which there are more than one unknown.

  18. Rob: I have read Jaynes book and enjoyed it. It dampened my desire to look at fuzzy logic etc. since it shows that Bayesian inference is the unique solution to a set of reasonable axioms.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>