A while back I wrote a post on three views of the negative binomial distribution. This post adds a fourth view.
One of the shortcomings of the Poisson distribution is that its variance exactly equals its mean. It is common in practice for the variance of count data to be larger than the mean, so it’s natural to look for a distribution like the Poisson but with larger variance. We start with a Poisson random variable X with mean λ, but then we make λ itself random and suppose that λ comes from a gamma(α, β) distribution. Then the marginal distribution on X is a negative binomial distribution with parameters r = α and p = 1/(β + 1).
The previous post said that the negative binomial is useful because it has more variance than the Poisson. The derivation above explains why the negative binomial should have more variance than the Poisson.
For details, see updated notes on the negative binomial distribution.
Slightly related comment:
A while ago I was looking for a distribution over discrete variables where I could control both the mean and the variance separately. As you clearly explained, the negative binomial only handles the overdispersed case, the Poisson distribution handles the case when mean and variance are equal and the binomial handles the underdispersed case.
I finally ran into this distribution: Conway-Maxwell-Poisson. This distribution allows one to control mean and variance directly. Interesting aside: it is an exponential family family distribution (which doesn’t make it easier to work with though).