Here’s a way to find a 95% confidence interval for any parameter θ.
- With probability 0.95, return the real line.
- With probability 0.05, return the empty set.
Clearly 95% of the time this procedure will return an interval that contains θ.
This example shows the difference between a confidence interval and a credible interval.
A 95% credible interval is an interval such that the probability is 95% that the parameter is in the interval. That’s what people think a 95% confidence interval is, but it’s not.
Suppose I give you a confidence interval using the procedure above. The probability that θ is in the interval is 1 if I return the real line and 0 if I return the empty set. In either case, the interval that I give you tells you absolutely nothing about θ.
But if I give you a 95% credible interval (a, b), then given the model and the data that went into it, the probability is 95% that θ is in the interval (a, b).
Confidence intervals are more useful in practice than in theory because they often approximately correspond to a credible interval under a reasonable model.
Credible intervals depend on your modeling assumptions. So do confidence intervals.