Comments on: Soft maximum

By: think again! » Blog Archive » Soft maximum

think again! » Blog Archive » Soft maximum — Sun, 14 Feb 2010 17:03:19 +0000

[...] Problem source: The blog of John D Cook. [...]

By: links for 2010-01-21 « Blarney Fellow

links for 2010-01-21 « Blarney Fellow — Fri, 22 Jan 2010 01:32:54 +0000

[...] Soft maximum — The Endeavour ~1min (tags: math optimization machine-learning) [...]

By: Tomasz Wegrzanowski

Tomasz Wegrzanowski — Thu, 21 Jan 2010 13:46:09 +0000

(someone asked about it on reddit, I can as well answer it here)

If you need a guarantee that \forall_i softmax(x1,…,xn) <= xi use this function instead: softmax(x1,…xn) = log(exp(x1) + … + exp(xn)) – log(n)

Proof: log(exp(x1) + … + exp(xn)) – log(n) <= log(n * exp(xmax)) – log(n) = log(n) + log(exp(xmax)) – log(n) = log(exp(xmax)) = xmax

Derivatives identical to the function from OP. Correction term changes depending on your choice of logarithm and exponentiation base.

And yes, the function will easily overflow when naively implemented with floating point units. Computing it as xmax + softmax(x1-xmax,…,xn-xmax) trivially solves this problem, as all arguments of exponentiation will be nonpositive, and argument of the logarithm will be between 1 and n.

By: Stephan Schmidt

Stephan Schmidt — Thu, 21 Jan 2010 06:01:57 +0000

@Tim: You can also use Wolfram Alpha

http://www.wolframalpha.com/input/?i=g(x,+y)+%3D+log(+exp(x)+%2B+exp(y)+)

By: John

John — Wed, 20 Jan 2010 23:33:51 +0000

Andrew: You're right that you'd need to be careful implementing g(x, y). The intermediate computations could easily overflow even though the final value would be a moderate-sized number. A robust implementation of g(x, y) would not just turn the definition into source code. See this post for one way to compute soft maximum while avoiding overflow.

By: Andrew Dalke

Andrew Dalke — Wed, 20 Jan 2010 23:19:25 +0000

Yes. That’s the same approach as we did in high school to have “the equation for a square: x**N + y**N = R**N where N is very large.” But then there’s numerical problems like k=8;log(exp(98*k) + exp(101.1*k))/k overflows a IEEE double. In any case, this is a trick I’m going to keep in mind for the future, so thanks for writing about it!

By: John

John — Wed, 20 Jan 2010 22:48:24 +0000

Andrew: You can get a better approximation by using a different kind of normalization. See the extra parameter k in g(x, y; k) defined near the end of the post.

By: The “Soft Maximum” function

The “Soft Maximum” function — Wed, 20 Jan 2010 22:44:52 +0000

[...] full post on Hacker News If you enjoyed this article, please consider sharing it! Tagged with: function [...]

By: Andrew Dalke

Andrew Dalke — Wed, 20 Jan 2010 22:38:13 +0000

Ahh, of course. But keep y small then my g(x,y) ~ log(exp(x)/2) = x – log(2) ~ x. In either case it’s a constant difference and it’s more a question of where you want the error to go. I’m thinking more though about the region where things are sharp. In that case x and y are about the same.

By: John

John — Wed, 20 Jan 2010 22:14:12 +0000

Troy: one reason to replace the hard maximum with the soft maximum would be to use optimization methods (e.g. Newton’s method) that require objective functions to be twice-differentiable. I’ve also heard that the soft maximum function comes up in electrical engineering problems with power calculations.

Tim: I made my plots using Mathematica. The colors were the defaults for contour plots.

Andrew: The normalization you propose would make g(x, x) = x, which sounds like a nice property to have. But the more important property is that g(x, y) approaches max(x, y) as either argument gets large. That property would no longer hold if you included the normalization constant.

By: Troy Gilbert

Troy Gilbert — Wed, 20 Jan 2010 22:10:50 +0000

Very interesting. I’m curious to know some examples of where and why you’d replace the hard maximum with the soft maximum?

By: Andrew Dalke

Andrew Dalke — Wed, 20 Jan 2010 21:53:34 +0000

You’ve left out a normalization, I think, since g(x,x) = log(exp(x)+exp(x)) = log(2*exp(x)) = log(2) + x, which is not f(x,x)= x. You likely want g(x,y) = log((exp(x)+exp(y))/2), which is the Generalized f-mean that Peter Turney mentions.

By: Tim

Tim — Wed, 20 Jan 2010 21:49:18 +0000

What software did you use to create this graph?

http://www.johndcook.com/contoursoft.png

I really like the color palette

By: Peter Turney

Peter Turney — Wed, 13 Jan 2010 15:02:13 +0000

The family of power means includes the arithmetic mean, the geometric mean, the harmonic mean, minimum, and maximum:

http://en.wikipedia.org/wiki/Power_means

By varying the exponent, you can make the “maximum” as soft or as hard as you want.

The soft maximum is a special case of the generalized f-mean:

http://en.wikipedia.org/wiki/Generalized_f-mean

By: Divye

Divye — Wed, 13 Jan 2010 13:58:58 +0000

Nice!

Found via your tweet.

By: Tomas Olsson

Tomas Olsson — Wed, 13 Jan 2010 13:47:12 +0000

Great post!
/Tomas

By: John

John — Wed, 13 Jan 2010 13:29:26 +0000

Thanks, Douglas. I fixed the typo.

By: Douglas

Douglas — Wed, 13 Jan 2010 13:26:05 +0000

I think the line:
g(x, y) = log( exp(x), exp(y) ).
should be
g(x, y) = log( exp(x) + exp(y) ).
?

By: Slavomir Kaslev

Slavomir Kaslev — Wed, 13 Jan 2010 13:18:16 +0000

Nice find. =-)

This is actually the core of Exponential Shadow Maps technique: www-flare.cs.ucl.ac.uk/staff/J.Kautz/publications/esm_gi08.pdf