Comments on: Soft maximum

By: New tech reports — The Endeavour

New tech reports — The Endeavour — Wed, 12 Oct 2011 13:00:43 +0000

[...] had a request to turn my blog posts on the soft maximum into a tech report, so here it [...]

By: Soft maximum 関数 « Everyday Pieces ::

Soft maximum 関数 « Everyday Pieces :: — Sat, 01 Oct 2011 15:19:03 +0000

[...] ここに出てる図とか見ると、直感的で分かりやすいかも。 [...]

By: John

John — Tue, 13 Sep 2011 19:24:57 +0000

Charles B: The reason CS uses “log” for natural logarithm is that math does too. Unfortunately, math has two conventions. Before calculus, log may mean logarithm base 10. After calculus, log means natural log. All advanced math uses log for natural logarithm, and that’s why computer languages do too. Sorry for the confusion.

By: Charles B.

Charles B. — Tue, 13 Sep 2011 19:05:23 +0000

One complaint: I got a bit confused by your use of CS notation – why don’t you write “ln” when you write equations? (If these snippets were formatted as code I wouldn’t gripe, but since they’re not they should follow math conventions, right? ;P)

By: John

John — Wed, 07 Sep 2011 22:21:09 +0000

I have not published any of this, but I could put it in a tech report and give you something to cite.

By: Jared H.

Jared H. — Wed, 07 Sep 2011 18:08:50 +0000

I was wondering if you have formally published this information in any books, conferences, or journals? If so, could you please forward me the citation information so that I can read the formal papers (and any additional analysis you have done with this equation)? I think this approximation may be helpful for a research project I am just starting, however, I’d rather cite a published paper that has ideally been peer-reviewed.

Thanks!

By: Ashish S.

Ashish S. — Thu, 21 Jul 2011 20:59:16 +0000

Very nice article. However, I was not entirely convinced that ” In the limit the derivative becomes infinite as the soft maximum converges to the hard maximum.” I wasn’t sure because the derivative on either side was finite so I was expecting the derivative to go to some finite value (based on the subgradient idea).

If my f(x)=|x|, then f(x) = max(x,-x) ~ log( exp(kx), exp(-kx) )/k. Then df/dx=tanh(kx). So when x->0 and k->inf we can make the limit go to any value but only within -1,1. Is this right?

By: How to compute the soft maximum « Data, Knowledge & Life

How to compute the soft maximum « Data, Knowledge & Life — Tue, 28 Sep 2010 21:46:52 +0000

[...] by Weiwei on 24/01/2010 by John D. Cook（博主注：John Cook最近在他的个人博客The Endeavour上撰写的关于计算soft [...]

By: think again! » Blog Archive » Soft maximum

think again! » Blog Archive » Soft maximum — Sun, 14 Feb 2010 17:03:19 +0000

[...] Problem source: The blog of John D Cook. [...]

By: links for 2010-01-21 « Blarney Fellow

links for 2010-01-21 « Blarney Fellow — Fri, 22 Jan 2010 01:32:54 +0000

[...] Soft maximum — The Endeavour ~1min (tags: math optimization machine-learning) [...]

By: Tomasz Wegrzanowski

Tomasz Wegrzanowski — Thu, 21 Jan 2010 13:46:09 +0000

(someone asked about it on reddit, I can as well answer it here)

If you need a guarantee that \forall_i softmax(x1,…,xn) <= xi use this function instead: softmax(x1,…xn) = log(exp(x1) + … + exp(xn)) – log(n)

Proof: log(exp(x1) + … + exp(xn)) – log(n) <= log(n * exp(xmax)) – log(n) = log(n) + log(exp(xmax)) – log(n) = log(exp(xmax)) = xmax

Derivatives identical to the function from OP. Correction term changes depending on your choice of logarithm and exponentiation base.

And yes, the function will easily overflow when naively implemented with floating point units. Computing it as xmax + softmax(x1-xmax,…,xn-xmax) trivially solves this problem, as all arguments of exponentiation will be nonpositive, and argument of the logarithm will be between 1 and n.

By: Stephan Schmidt

Stephan Schmidt — Thu, 21 Jan 2010 06:01:57 +0000

@Tim: You can also use Wolfram Alpha

http://www.wolframalpha.com/input/?i=g(x,+y)+%3D+log(+exp(x)+%2B+exp(y)+)

By: John

John — Wed, 20 Jan 2010 23:33:51 +0000

Andrew: You're right that you'd need to be careful implementing g(x, y). The intermediate computations could easily overflow even though the final value would be a moderate-sized number. A robust implementation of g(x, y) would not just turn the definition into source code. See this post for one way to compute soft maximum while avoiding overflow.

By: Andrew Dalke

Andrew Dalke — Wed, 20 Jan 2010 23:19:25 +0000

Yes. That’s the same approach as we did in high school to have “the equation for a square: x**N + y**N = R**N where N is very large.” But then there’s numerical problems like k=8;log(exp(98*k) + exp(101.1*k))/k overflows a IEEE double. In any case, this is a trick I’m going to keep in mind for the future, so thanks for writing about it!

By: John

John — Wed, 20 Jan 2010 22:48:24 +0000

Andrew: You can get a better approximation by using a different kind of normalization. See the extra parameter k in g(x, y; k) defined near the end of the post.

By: The “Soft Maximum” function

The “Soft Maximum” function — Wed, 20 Jan 2010 22:44:52 +0000

[...] full post on Hacker News If you enjoyed this article, please consider sharing it! Tagged with: function [...]

By: Andrew Dalke

Andrew Dalke — Wed, 20 Jan 2010 22:38:13 +0000

Ahh, of course. But keep y small then my g(x,y) ~ log(exp(x)/2) = x – log(2) ~ x. In either case it’s a constant difference and it’s more a question of where you want the error to go. I’m thinking more though about the region where things are sharp. In that case x and y are about the same.

By: John

John — Wed, 20 Jan 2010 22:14:12 +0000

Troy: one reason to replace the hard maximum with the soft maximum would be to use optimization methods (e.g. Newton’s method) that require objective functions to be twice-differentiable. I’ve also heard that the soft maximum function comes up in electrical engineering problems with power calculations.

Tim: I made my plots using Mathematica. The colors were the defaults for contour plots.

Andrew: The normalization you propose would make g(x, x) = x, which sounds like a nice property to have. But the more important property is that g(x, y) approaches max(x, y) as either argument gets large. That property would no longer hold if you included the normalization constant.

By: Troy Gilbert

Troy Gilbert — Wed, 20 Jan 2010 22:10:50 +0000

Very interesting. I’m curious to know some examples of where and why you’d replace the hard maximum with the soft maximum?

By: Andrew Dalke

Andrew Dalke — Wed, 20 Jan 2010 21:53:34 +0000

You’ve left out a normalization, I think, since g(x,x) = log(exp(x)+exp(x)) = log(2*exp(x)) = log(2) + x, which is not f(x,x)= x. You likely want g(x,y) = log((exp(x)+exp(y))/2), which is the Generalized f-mean that Peter Turney mentions.

By: Tim

Tim — Wed, 20 Jan 2010 21:49:18 +0000

What software did you use to create this graph?

http://www.johndcook.com/contoursoft.png

I really like the color palette

By: Peter Turney

Peter Turney — Wed, 13 Jan 2010 15:02:13 +0000

The family of power means includes the arithmetic mean, the geometric mean, the harmonic mean, minimum, and maximum:

http://en.wikipedia.org/wiki/Power_means

By varying the exponent, you can make the “maximum” as soft or as hard as you want.

The soft maximum is a special case of the generalized f-mean:

http://en.wikipedia.org/wiki/Generalized_f-mean

By: Divye

Divye — Wed, 13 Jan 2010 13:58:58 +0000

Nice!

Found via your tweet.

By: Tomas Olsson

Tomas Olsson — Wed, 13 Jan 2010 13:47:12 +0000

Great post!
/Tomas

By: John

John — Wed, 13 Jan 2010 13:29:26 +0000

Thanks, Douglas. I fixed the typo.

By: Douglas

Douglas — Wed, 13 Jan 2010 13:26:05 +0000

I think the line:
g(x, y) = log( exp(x), exp(y) ).
should be
g(x, y) = log( exp(x) + exp(y) ).
?

By: Slavomir Kaslev

Slavomir Kaslev — Wed, 13 Jan 2010 13:18:16 +0000

Nice find. =-)

This is actually the core of Exponential Shadow Maps technique: www-flare.cs.ucl.ac.uk/staff/J.Kautz/publications/esm_gi08.pdf