There’s a theorem in statistics that says
You could read this aloud as “the mean of the mean is the mean.” More explicitly, it says that the expected value of the average of some number of samples from some distribution is equal to the expected value of the distribution itself. The shorter reading is confusing since “mean” refers to three different things in the same sentence. In reverse order, these are:
- The mean of the distribution, defined by an integral.
- The sample mean, calculated by averaging samples from the distribution.
- The mean of the sample mean as a random variable.
The hypothesis of this theorem is that the underlying distribution has a mean. Lets see where things break down if the distribution does not have a mean.
It’s tempting to say that the Cauchy distribution has mean 0. Or some might want to say that the mean is infinite. But if we take any value to be the mean of a Cauchy distribution — 0, ∞, 42, etc. — then the theorem above would be false. The mean of n samples from a Cauchy has the same distribution as the original Cauchy! The variability does not decrease with n, as it would with samples from a normal, for example. The sample mean doesn’t converge to any value as n increases. It just keeps wandering around with the same distribution, no matter how large the sample. That’s because the mean of the Cauchy distribution simply doesn’t exist.
The Cauchy distribution has many more interesting properties. It also has no variance defined, does not satisfy the classical central limit theorem nor the strong law of large numbers. It is strictly stable.
Take a random variable distributed uniformly on the unit square, i.e. two independent uniform random variables on the interval (-1,1). Call them X and Y. Then the distribution of W:=X/Y is Cauchy. When Y is near 0, we have W to be large, which can be used to explain the heavy tails of the Cauchy distribution.
Is this what they call the Central Limit Theorem?
No, I don’t know if this theorem has a name. But the Central Limit Theorem says that the average of a large number of random variables converges to a Gaussian distribution. It’s enough to assume that the random variables are independent, have the same distribution, and have finite variance.