Differential equations

I studied nonlinear PDEs in grad school. My advisor, Ralph Showalter, would remind us occasionally what ‘nonlinear’ means.

“Nonlinear” is not a hypothesis but the lack of a hypothesis.

He meant a couple things by this. First, when people say “nonlinear,” they often mean “not necessarily linear.” That is, they use “nonlinear” as a generalization of linear. If a statement doesn’t hold for linear equations, it can’t hold more generally. So try the linear case first.

Second, and more importantly, you usually have to specify in what way an equation is nonlinear before you can say anything useful. If you’re not assuming linearity, what are you assuming? Maybe you need to assume a function is convex. Or maybe you need to assume an upper or lower bound on a function’s growth. In any case, focus on what you are assuming rather than what you are not assuming, and make your assumptions explicit.

Related post: Three views of differential equations

I’ve started to read through Michael Fogus’ list of recommended technical papers and ran across this gem from Out of the Tar Pit:

Complexity breeds complexity. There are a whole set of secondary causes of complexity. This covers all complexity introduced as a result of not being able to clearly understand a system. Duplication is a prime example of this — if (due to state, control or code volume) it is not clear that functionality already exists, or it is too complex to understand whether what exists does exactly what is required, there will be a strong tendency to duplicate. This is particularly true in the presence of time pressures.

Within a few hours of reading that quote I had a conversation illustrating it. I talked with someone who needed to make a small change to a complex section of code. He said the code had three minor variations on the same chunk of functionality. He could get his job done much faster in the short term if he simply added a forth mutation to the code base. He refused to do that, but many developers would not refuse.

Suppose the rate of growth in complexity of a project is proportional to how complex the project is. And suppose, as the quote above suggests, that the proportionality constant is the time pressure. Then the complexity over time is given by

y‘(t) = a y(t)

where y(t) is complexity at time t and a is the time pressure. Then complexity grows exponentially. The solution to the equation is

y(t) = y₀ e^at

where y₀ is the initial complexity. This isn’t meant to be an exact model, just a back-of-the-envelope illustration. On the other hand, I’ve seen situations where it gives a fairly good description of a project for a while. Complexity can grow exponentially like compound interest, and the greater the pressure, the greater the rate of compounding.

Now suppose there’s a different kind of time pressure, a pressure to simplify a project. This would correspond to a negative value of the proportionality constant a. If there were such pressure, this would mean that complexity would decrease exponentially.

I don’t think this kind of negative pressure on complexity is as realistic as positive pressure, but it’s not entirely unrealistic either. In the rare case of pressure to simplify, removing one source of complexity could lead to cascading effects. Because we don’t need this one thing any more, we don’t need these other things that were only there to prop it up, etc. There could be a sustained decrease in complexity, though it probably would not be exponential.

Related post:

A little simplification goes a long way

The most common view of differential equations may be sheer terror, but those who get past terror may have one of the following perspectives.

Naive view: All differential equations can be solved in closed form by applying one of the 23 tricks covered in your text book.

Sophomoric view: Differential equations that come up in practice can almost never be solved in closed form, so it’s not worth trying. Learn numerical techniques and don’t bother with analytic solutions.

Classical view: Some very important differential equations can be solve in closed form, especially if you expand your definition of “closed form” to include a few special functions.  Analytic solutions to these equations will tell you things that would be hard to discover from numerical solutions alone.

***

I never held the naive view; I learned the sophomoric view before I knew much about differential equations. There’s a lot of truth in the sophomoric view — that’s why it’s called sophomoric.  It’s not entirely wrong, it’s just incomplete. (More on that below.)

I’ve learned differential equations in a sort of reverse-chronological order. I learned the modern theory first — existence and uniqueness theorems, numerical techniques, etc. — and only learned the classical theory much later. I studied nonlinear PDEs before knowing much about linear PDEs.  This may be the most efficient way to learn, begin with the end in mind and all that. It almost certainly is the fastest way to get out of graduate school. But it’s not very satisfying.

***

I get in trouble whenever I mention etymologies. So at the risk of sounding like Gus Portokalos from My Big Fat Greek Wedding, I’ll venture another etymology. I’ve always heard that sophomore comes from the Greek words sophos (wise) and moros (fool), though something I read suggested this may be a folk etymology. It doesn’t matter: regardless of whether that is the correct historical origin of the word, it accurately conveys the sense of the word. The idea is that a sophomore has learned a little knowledge but is over-confident in that knowledge and doesn’t know its boundaries. In mathematical terms, it’s someone who has learned a first-order approximation to the truth and extrapolates that approximation too far.

***

Related posts:

Grand unified theory of classical math
Approximating a solution that doesn’t exist

The heart of 19th century math was the study of special functions arising from mathematical physics.

It is well known that the central problem of the whole of modern mathematics is the study of the transcendental functions defined by differential equations.

The above quote was the judgment of  Felix Klein (of Klein bottle fame) in 1893. The differential equations he had in mind were the second order differential equations of mathematical physics.

Special functions were the core of 19th century math, and hypergeometic series were the grand unifying theory of special functions. (Not every special function is hypergeometric, but quite a few are.) And yet they’re hardly taught any more. I never heard of hypergeometric series in college, even though I studied differential equations and applied math. Later I encountered hypergeometric functions first in combinatorics and only later in differential equations.

It’s odd that what was “the central problem of the whole of modern mathematics” could become almost a lost art a century later. How could this be? I believe part of the explanation is that special functions, and hypergeometric function in particular, fall between two stools: too advanced for undergraduate programs but not a hot enough of a research area for graduate programs.

When I was a little kid, I asked some adults the following question.

If hot things cool, and cool things warm up, could something hot cool down and warm back up?

The people I asked didn’t understand my question and just laughed. I have no idea how old I was, but I wasn’t old enough to articulate what I was thinking.

Here’s what I had in mind. I knew that hot things like a cup of coffee grew cold. And I knew that cold things, say a glass of milk, get warm. Well, could the coffee get so cold that it becomes a cold thing and start to warm back up?

Could the coffee become as cold as the glass of milk? Common sense suggests that can’t happen. When we say coffee grows cold, we mean that it becomes relatively colder, closer to room temperature. And when we say the milk is getting warm, we also mean it is getting closer to room temperature. We’ve never left a hot cup of coffee on a table and come back later to find that it has cooled off so much that it is colder than room temperature. But could there be small fluctuations?

As the coffee and milk head toward room temperature, could they overshoot the target, just by a little bit? Say room temperature is 70 °F, the coffee starts out at 150 °F, and the milk starts out at 40 °F. We don’t expect the coffee to cool down to 40 °F or the milk to warm up to 150 °F. But could the coffee cool down to 69.5 °F and then go back up to 70 °F? Could the milk warm up to 70.5 °F and then cool back down to 70 °F?

I didn’t get a satisfactory answer to my childhood question until I was in college. Then I found out about Newton’s law of cooling. It says that the rate at which a warm body cools is proportional to the difference between its current temperature and the ambient temperature. This law can be written as a differential equation whose solution shows that the temperature of a warm body decreases exponentially to the ambient temperature. The temperature curve always slopes downward. It doesn’t wiggle even a little on its journey to room temperature. Cold bodies warm up the opposite way, exponentially approaching room temperature but never exceeding it.

In case this seems obvious, think about thermostats. They don’t work this way. Say the temperature in a room is 85 °F and you’d like it to be 72 °F, so you turn on the air conditioning. Will the temperature steadily lower to 72 °F? Not exactly. If you were to plot the temperature in the room over time and look at the graph from far enough away, it would look like it is steadily going down to the desired temperature. But if you look at the graph more closely, you’ll see wiggles. The AC may cool the room to a little below 72 °F, maybe to 70 °F. The AC would cut off and the temperature would rise to 72 °F. Unlike the cup of hot coffee, the AC will often overshoot its target, though not by too much. The temperature may feel constant, but it is not. It oscillates around the desired temperature.

How can we extend the idea of derivative so that more functions are differentiable? Why would we want to do so? How can we make sense of a delta “function” that isn’t really a function? We’ll answer these questions in this post.

[click to continue...]

The following example made an impression on me when I first saw it years ago. I still think it’s an important example, though I’d draw a different conclusion from it today.

Problem: Let y(t) be the solution to the differential equation y’ = t² + y² with y(0) = 1. Calculate y(1).

If we use Euler’s numerical method with a step size h = 0.1, we get y(1) = 7.19. If we reduce the step size to 0.05 we get y(1) = 12.32. If we reduce the step size further to 0.01, we get y(1) = 90.69. That’s strange. Let’s switch over to a more accurate method, Runge-Kutta. With a step size 0.1 the Runge-Kutta method gives 735.00, and if we use a step size of 0.01 we get a result larger than 10¹⁵. What’s going on?

The problem presupposes that a solution exists at t = 1 when in fact no solution exists. General theory (Picard’s theorem) tells that a unique solution exists for some interval containing 0, but it does not tell us how far that interval extends. With a little work we can show that a solution exists for t at least as large as π/4. However, the solution becomes unbounded somewhere between π/4 and 1.

When I first saw this example, my conclusion was that it showed how important theory is. If you just go about numerically computing solutions without knowing that a solution exists, you can think you have succeeded when you’re actually computing something that doesn’t exist. Prove existence and uniqueness before computing. Theory comes first.

Now I think the example shows the importance of the interplay between theory and numerical computation. It would be nice to know how big the solution interval is before computing anything, but that’s not always possible. Also, it’s not obvious from looking at the equation that there should be a problem at t = 1. The difficulties we had with numerical computation suggested there might be a theoretical problem.

I first saw this problem in an earlier edition of Boyce and DiPrima. The book goes on to approximate the interval over which the solution does exist using a combination of analytical and numerical methods. It looks like the solution becomes unbounded somewhere near t = 0.97.

I wouldn’t say that theory or computation necessarily come first. I’d say you iterate between them, starting with the approach that is more tractable. Theoretical results are  more satisfying when they’re available, but theory often doesn’t tell us as much as we’d like to know. Also, people make mistakes in theoretical computation just as they do in numerical computation. It’s best when theory and numerical work validate each other.

The problem does show the importance of being concerned with existence and uniqueness, but theoretical methods are not the only methods for exploring existence. Good numerical practice, i.e. trying more than one step size or more than one numerical method, is also valuable. In any case, the problem shows that without some diligence — either theoretical or numerical — you could naïvely compute an approximate “solution” where no solution exists.