Uniformity increases entropy

Suppose you have a system with n possible states. The entropy of the system is maximized when all states are equally likely to occur. The entropy is minimized when one outcome is certain to occur.

You can say more. Starting from any set of probabilities, as you move in the direction of more uniformity, you increase entropy. And as you move in the direction of less uniformity, you decrease entropy.

These statements can be quantified and stated more precisely. That’s what the rest of this post will do.

***

Let p_i be the probability of the ith state and let p be the vector of the p_i.

$\mathbf{p} = (p_1, p_2, \ldots, p_n)$

Then the entropy of p is defined as

$H(\mathbf{p}) = -\sum_{i=1}^n p_i \log_2 p_i$

If one of the p‘s is 1 and the rest of the p‘s are zero, then H(p) = 0. (In the definition of entropy, 0 log₂ 0 is taken to be 0. You could justify this as the limit of x log₂ x as x goes to zero.)

If each of the p_i are equal, p_i = 1/n, then H(p) = log₂ n. The fact that this is the maximum entropy, and that compromises between the two extremes always decrease entropy, comes from the fact that the entropy function H is concave (proof). That is, if p₁ is one list of probabilities and p₂ another, then

$H((1-\lambda) \mathbf{p}_1 + \lambda\mathbf{p}_2) \geq (1 - \lambda) H( \mathbf{p}_1) + \lambda H(\mathbf{p}_2)$

When we speak informally of moving from p₁ in the direction of p₂, we mean we increase the parameter λ from 0 to some positive amount no more than 1.

Because entropy is concave, there are no local maxima. As you approach the location of global maximum entropy, i.e. equal state probabilities, from any direction, entropy increases monotonically.

Leave a Reply