History

The Central Limit Theorem says that if you average enough independent copies of a random variable, the result has a nearly normal (Gaussian) distribution. Of course that’s a very rough statement of the theorem. What are the precise requirements of the theorem? That question took two centuries to resolve. You can see the final answer here.

The first version of the Central Limit Theorem appeared in 1733, but necessary and sufficient conditions weren’t known until 1935. I won’t recap the entire history here. I just want to comment briefly on how the Central Limit Theorem began and how different the historical order of events was from the typical order of presentation.

A typical probability course might proceed as follows.

1. Define the normal distribution.
2. State and prove a special case of the Central Limit Theorem.
3. Present the normal approximation to the binomial as a corollary.

This is the opposite of the historical order of events.

Abraham de Moivre discovered he could approximate binomial distribution probabilities using the integral of exp(-x²) and proved an early version of the Central Limit Theorem in 1733. At the time, there was no name given to his integral. Only later did anyone think of exp(-x²) as the density of a probability distribution. De Moivre certainly didn’t use the term “Gaussian” since Gauss was born 44 years after de Moivre’s initial discovery. De Moivre also didn’t call his result the “Central Limit Theorem.” George Pólya gave the theorem that name in 1920 as it was approaching its final form.

For more details, see The Life and Times of the Central Limit Theorem.

Related links:

Sums of uniform random variables
Quantifying the error in the central limit theorem
Three central limit theorems

Today is Cinco de Mayo, the holiday that celebrates the Mexican army’s defeat of French forces at the Battle of Puebla on May 5, 1862.

Cinco de Mayo is unusual in that it is a Mexican holiday more popular in the United States than in Mexico. According to Wikipedia,

While Cinco de Mayo has limited or no significance nationwide in Mexico, the date is observed in the United States and other locations around the world as a celebration of Mexican heritage and pride.

Cinco de Mayo is a bigger holiday in Texas than Texas Independence Day. (Readers unfamiliar with Texas history may be surprised to learn that Texas was once a sovereign nation. The Republic of Texas existed for nearly a decade between gaining independence from Mexico in 1836 and joining the United States in 1845.)

Texas Independence Day, March 2, usually goes virtually unnoticed. However in 1986, the sesquicentennial, there was a big celebration in Austin. Activities included baking the world’s largest cake. The left-overs were distributed to the dorms at the University of Texas and so I had some of the cake. Quite a bit, actually. You might think that a cake baked for the purpose of setting a world record would be barely edible, but it was actually pretty good lemon cake.

Innovation is not the same as invention. According to Peter Denning,

An innovation is a transformation of practice in a community. It is not the same as the invention of a new idea or object. The real work of innovation is in the transformation of practice. … Many innovations were preceded or enabled by inventions; but many innovations occurred without a significant invention.

Michael Schrage makes a similar point.

I want to see the biographies and the sociologies of the great customers and clients of innovation. Forget for awhile about the Samuel Morses, Thomas Edisons, the Robert Fultons and James Watts of industrial revolution fame. Don’t look to them to figure out what innovation is, because innovation is not what innovators do but what customers adopt.

Innovation in the sense of Denning and Schrage is harder than invention. Most inventions don’t lead to innovations.

The simplest view of the history of invention is that Morse invented the telegraph, Fulton the steamboat, etc. A sophomoric view is that men like Morse and Fulton don’t deserve so much credit because they only improved on and popularized the inventions of others. A more mature view is that Morse and Fulton do indeed deserve the credit they receive. All inventors build on the work of predecessors, and popularizing an invention (i.e. encouraging innovation) requires persistent hard work and creativity.

The normal distribution pops up everywhere in statistics. Contrary to popular belief, the name does not come from “normal” as in “conventional.” Instead the term comes from a detail in a proof by Gauss discussed below where he showed that two things were perpendicular in a sense.

(The word “normal” originally meant “at a right angle,” going back to the Latin word normalis for a carpenter’s square. Later the word took on the metaphorical meaning of something in line with custom. Mathematicians sometimes use “normal” in the original sense of being orthogonal.)

The mistaken etymology persists because the normal distribution is conventional. Statisticians often assume anything random has a normal distribution by default. While this assumption is not always justified, it often works remarkably well. This post gives four lines of reasoning that lead naturally to the normal distribution.

1) The earliest characterization of the normal distribution is the central limit theorem, going back to Abraham de Moivre. Roughly speaking, this theorem says that if you average enough distributions together, even if they’re not normal, in the limit their average is normal. But this justification for assuming normal distributions everywhere has a couple problems. First, the convergence in the central limit theorem may be slow, depending on what is being averaged. Second, if you relax the hypotheses on the central limit theorem, other stable distributions with thicker tails also satisfy a sort of central limit theorem. The characterizations given below are more satisfying because they do not rely on limit theorems.

2) The astronomer William Herschel discovered the simplest characterization of the normal. He wanted to characterize the errors in astronomical measurements. He assumed (1) the distribution of errors in the x and y directions must be independent, and (2) the distribution of errors must be independent of angle when expressed in polar coordinates. These are very natural assumptions for an astronomer, and the only solution is a product of the same normal distribution in x and y. James Clerk Maxwell came up with an analogous derivation in three dimensions when modelling gas dynamics.

3) Carl Friedrich Gauss came up with the characterization of the normal distribution that caused it to be called the “Gaussian” distribution. There are two strategies for estimating the mean of a random variable from a sample: the arithmetic mean of the samples, and the maximum likelihood value. Only for the normal distribution do these coincide.

4) The final characterization listed here is in terms of entropy. For a specified mean and variance, the probability density with the greatest entropy (least information) is the normal distribution. I don’t know who discovered this result, but I read it in C. R. Rao’s book. Perhaps it’s his result. If anyone knows, please let me know and I’ll update this post. For advocates of maximum entropy this is the most important characterization of the normal distribution.

Related post:

How the Central Limit Theorem began

During the Victorian era, Scotland produced the best engineers in the world. It became routine for British ships to have a Scottish engineer on board. Star Trek’s Scottish engineer Montgomery Scott reflects this tradition.

Source: Victorian Britain