More theoretical power, less real power

Suppose you’re deciding between two statistical methods. You pick the one that has more power. This increases your chances of making a correct decision in theory while possibly lowering your chances of actually concluding the truth. The subtle trap is that the meaning of “in theory” changes because you have two competing theories.

When you compare the power of two methods, you’re evaluating each method’s probability of success under its own assumptions. In other words, you’re picking the method that has the better opinion of itself. Thus the more powerful method is not necessarily the method that has the better chance of leading you to a correct conclusion.

Comparing power alone is not enough. You also need to evaluate whether a method makes realistic assumptions and whether it is robust to deviations from its assumptions.

Related posts:

Most published research results are false

Canonical examples from robust statistics

Tagged with:
Posted in Statistics
6 comments on “More theoretical power, less real power
  1. Can you illustrate this with a concrete example? I think I understand the idea, but…

  2. Q says:

    In almost all scenarios in science the strict null hypothesis is false, so the more powerful statistical method should be used. Of more importance is the effect size, often best illustrated using confidence intervals.

  3. John says:

    Eugene: Here’s an example. Consider parametric versus non-parametric tests. The former make stronger distributional assumptions, such as assuming data come from a normal distribution. Often a parametric test has more power than its non-parametric analog. That means the parametric test is more likely to find statistical significance if its distributional assumptions are correct. But if the distributional assumptions are not satisfied, the non-parametric test may be more powerful.

    People think they’re placing one bet when they’re actually placing two bets. They’re betting on their modeling assumptions being correct, and betting on their method working given the modeling assumptions. Too often only the latter comes to mind.

    Q: I agree that strict null hypotheses are almost certainly false, but I’ll leave that aside. Let’s say you want to estimate effect size with a confidence interval as you suggest. Then the smaller interval is better. Assume your data come from a normal distribution. If also you assume you know the variance, you can get a smaller confidence interval than if you acknowledge that you are only estimating the variance. So by making a stronger (unjustified) assumption, you can get “better” results. Your understanding of reality hasn’t improved, but your numbers have. That’s an elementary example, and not too many people would fall for that. But people do analogous things in more subtle circumstances.

  4. “People think they’re placing one bet when they’re actually placing two bets….”

    Thanks–your example of parametric tests was just what I needed.

  5. MathDr says:

    Why would you have the method be robust against deviations of its assumptions? The method is (usually) designed for use when the assumptions are valid. One should have another method that selects between the methods when the assumptions are different, right?

  6. John says:

    MathDr: Because your assumptions are never entirely valid.

    This is a statistical analog to physicists wondering how sensitive a system is to initial conditions. Do small changes in initial conditions lead to small changes in final conditions as in stable systems, or can small changes in initial conditions have enormous consequences as in chaotic systems? You need to know which realm you’re in.

3 Pings/Trackbacks for "More theoretical power, less real power"
  1. [...] This post was mentioned on Twitter by John D. Cook, warren. warren said: RT @JohnDCook: New post: More theoretical power, less real power http://bit.ly/fsLzvr #statistics [...]

  2. [...] More theoretical power, less real power Cost-benefit analysis versus benefit-only analysis ? X [...]

  3. [...] More theoretical power, less real power Advantages of crude models Canonical examples from robust statistics [...]