Multiple comparisons present a conundrum in classical statistics. The options seem to be:
- do nothing and tolerate a high false positive rate
- be extremely conservative and tolerate a high false negative rate
- do something ad hoc between the extremes
A new paper by Andrew Gelman, Jennifer Hill, and Masanao Yajima opens with “The problem of multiple comparisons can disappear when viewed from a Bayesian perspective.” I would clarify that the resolution comes not from the Bayesian perspective per se but from the Bayesian hierarchical perspective.
See this blog post for a link to the article “Why we (usually) don’t have to worry about multiple comparisons” and to a presentation by the same title.