Here’s an interesting problem that came out of a logistic regression application. The input variable was between 0 and 1, and someone asked when and where the logistic transformation
f(x) = 1/(1 + exp(a + bx))
has a fixed point, i.e. f(x) = x.
So given logistic regression parameters a and b, when does the logistic curve given by y = f(x) cross the line y = x? Do they always cross? Can they cross twice?
There’s always at least one solution. Because f(x) is strictly between 0 and 1, the function
g(x) = f(x) − x
is positive at 0 and negative at 1, and by the intermediate value theorem g(x) must be zero for some x between 0 and 1.
Sometimes f has only one fixed point. It may have two or three fixed points, as demonstrated in the graph below. The case of two fixed points is unstable: the logistic curve is tangent to the line y = x at one point, and a tiny change would turn this tangent point into either no crossing or two crossings.
If |b| < 1, then you can show that the function f is a contraction map on [0, 1]. In that case there is a unique solution to f(x) = x, and you can find it by starting with an arbitrary value for x and repeatedly applying f to it. For example, if a = 1 and b = 0.8 and we start with x = 0, after applying f ten times we get x = f(x) = 0.233790157.
There are a couple questions left to finish this up. How can you tell from a and b how many fixed points there will be? The condition |b| < 1 is sufficient for f to be a contraction map on [0, 1]. Can you find necessary and sufficient conditions?
Related post: Sensitivity of logistic regression prediction
The second question, at least, is reasonably easy to answer. Since f is differentiable, we can use its derivative to determine whether or not it is a contraction map. We have f'(x) = -b exp(a + bx) / (1 + exp(a + bx))^2. If we let u = exp(a + bx), then this is -b u / (1 + u^2), which is maximized (in absolute value) at b/2 when u = 1. Hence, f is a contraction on the real numbers if |b| 2. If |b| = 2, then f is a weak contraction.
But wait! Since f takes values in [0, 1], we might as well consider it as a function on [0, 1] when looking for fixed points. There are three cases:
1) u = 1 for some x in [0, 1]. This happens when one of a and a + b is positive, and the other is negative. The maximum value of f’ occurs somewhere in [0, 1], so the picture is the same as when considering f on the real numbers.
2) u < 1 for all x in [0, 1]. This happens when both a and a + b are negative. The maximum value of |f'| on [0, 1] occurs when u is maximized, which is at x = 0 if b is negative, or x = 1 if b is positive. If we calculate f' as above at this value, then f is a contraction if |f'| 1, and a weak contraction if |f’| = 1. Because [0, 1] is compact, a weak contraction is as good as a contraction for the purpose of finding fixed points.
3) u > 1 for all x in [0, 1]. This happens when both a and a + b are positive. Same as case 2, except the maximum value of |f’| is when u is minimized.
Nathan, I think you have misplaced the exponent (2):
if u=exp(a + bx), then f'(x) = -b u / (1 + u)^2, so the maximal value of |f’| is |b|/4, at u=1. Or in an other way: f'(x)= -b exp(a + bx) / (1 + exp(a + bx))^2 = -b f(x) (1-f(x)). Since 0< f(x) < 1, then f(x)(1-f(x)) reaches its maximum at f(x)=1/2, which is 1/4. So if |b| < 4, then f is a contraction.