Here are some of my favorite notations that are not commonly used.
The first is Richard Stanley’s notation for counting the number of ways to select k objects from a set of n objects with replacement. This is similar to the problem solved by binomial coefficients, but not the same since binomial coefficients count the number of possible selections without replacement. Stanley’s symbol is
I like this symbol for two reasons. First, it’s good to have a notation, any notation, for a concept that comes up fairly often. Second, it’s appropriate for this symbol to resemble the binary coefficient symbol. See selecting with replacement for more on Stanley’s symbol, how to think about it and how to compute it.
Next is Iverson’s notation for indicator functions. Iverson’s idea was to put a Boolean condition in square brackets to indicate the function that is 1 when that condition is true and 0 otherwise. For example, [x > y] is the function f(x, y) such that f equals 1 when x is greater than y and equals 0 for all other arguments. This notation saves ink and makes it easier to concentrate on the substance of an expression. For more on Iverson’s notation, see Concrete Mathematics.
Another notation from Concrete Mathematics is the use of a perpendicular symbol to note that two integers are relatively prime. For example, m ⊥ n would indicate that that m and n are relatively prime. The more common way to denote this would be to say gcd(m, n) = 1. The perpendicular symbol is nice because perpendicular lines have no component of direction in common, just as relative prime numbers have no prime factors in common.
Finally, multi-index notation is a handy way to make multivariable theorems easier to remember. For example, with this notation, Taylor series in several variables look similar to Taylor series in one variable.
Related link: Stanley’s twelvefold way